[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:31:54 dist-test-slave-cbn3 krb5kdc[10771](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:31:54 dist-test-slave-cbn3 krb5kdc[10771](info): set up 2 sockets
Jun 20 18:31:54 dist-test-slave-cbn3 krb5kdc[10771](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 18:31:55.222506 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.003s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:31:55 dist-test-slave-cbn3 krb5kdc[10771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:31:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:31:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.131.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
I20260620 18:31:55.399096 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:41221
--webserver_interface=127.10.131.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:41221
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:31:55.725646 10784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:31:55.726040 10784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:31:55.726253 10784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:31:55.735101 10784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:31:55.735227 10784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:31:55.735328 10784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:31:55.735450 10784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:31:55.735544 10784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:31:55.746783 10784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:41221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:41221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:31:55.748265 10784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:31:55.749806 10784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:31:55.763072 10792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:31:55.763120 10790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:31:55.763125 10789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:31:55.766336 10784 server_base.cc:1034] running on GCE node
I20260620 18:31:55.767138 10784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:31:55.768680 10784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:31:55.769878 10784 hybrid_clock.cc:648] HybridClock initialized: now 1781980315769848 us; error 50 us; skew 500 ppm
I20260620 18:31:55.770236 10784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:31:55 dist-test-slave-cbn3 krb5kdc[10771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980315, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:31:55.773440 10784 init.cc:377] Logged in from keytab as kudu/127.10.131.62@KRBTEST.COM (short username kudu)
I20260620 18:31:55.774966 10784 webserver.cc:460] Webserver started at http://127.10.131.62:33745/ using document root <none> and password file <none>
I20260620 18:31:55.775700 10784 fs_manager.cc:359] Metadata directory not provided
I20260620 18:31:55.775827 10784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:31:55.776136 10784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:31:55.779153 10784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "c4e9126f43d649ccae0f72de158b91d9"
format_stamp: "Formatted at 2026-06-20 18:31:55 on dist-test-slave-cbn3"
server_key: "64a11b9ea55a32b8598e36b5bdfea777"
server_key_iv: "90b9d872d50f4b663ae1fa1d4f3d2d38"
server_key_version: "encryptionkey@0"
I20260620 18:31:55.779903 10784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "c4e9126f43d649ccae0f72de158b91d9"
format_stamp: "Formatted at 2026-06-20 18:31:55 on dist-test-slave-cbn3"
server_key: "64a11b9ea55a32b8598e36b5bdfea777"
server_key_iv: "90b9d872d50f4b663ae1fa1d4f3d2d38"
server_key_version: "encryptionkey@0"
I20260620 18:31:55.785091 10784 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 18:31:55.788497 10799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:31:55.789932 10784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:31:55.790098 10784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "c4e9126f43d649ccae0f72de158b91d9"
format_stamp: "Formatted at 2026-06-20 18:31:55 on dist-test-slave-cbn3"
server_key: "64a11b9ea55a32b8598e36b5bdfea777"
server_key_iv: "90b9d872d50f4b663ae1fa1d4f3d2d38"
server_key_version: "encryptionkey@0"
I20260620 18:31:55.790283 10784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:31:55.813653 10784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:31:55.818004 10784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:31:55.818302 10784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:31:55.836989 10784 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:41221
I20260620 18:31:55.837036 10850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:41221 every 8 connection(s)
I20260620 18:31:55.838356 10784 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:31:55.842113 10851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:31:55.846335 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 10784
I20260620 18:31:55.846693 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:31:55.849268 10764 external_mini_cluster.cc:1428] Setting key 4e8b31b48f70189273a41c9f97d48d5d
I20260620 18:31:55.855552 10851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4e9126f43d649ccae0f72de158b91d9: Bootstrap starting.
I20260620 18:31:55.858793 10851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4e9126f43d649ccae0f72de158b91d9: Neither blocks nor log segments found. Creating new log.
I20260620 18:31:55.859838 10851 log.cc:824] T 00000000000000000000000000000000 P c4e9126f43d649ccae0f72de158b91d9: Log is configured to *not* fsync() on all Append() calls
I20260620 18:31:55.862666 10851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4e9126f43d649ccae0f72de158b91d9: No bootstrap required, opened a new log
I20260620 18:31:55.873339 10851 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4e9126f43d649ccae0f72de158b91d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e9126f43d649ccae0f72de158b91d9" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 41221 } }
I20260620 18:31:55.873839 10851 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4e9126f43d649ccae0f72de158b91d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:31:55.873960 10851 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4e9126f43d649ccae0f72de158b91d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4e9126f43d649ccae0f72de158b91d9, State: Initialized, Role: FOLLOWER
I20260620 18:31:55.874565 10851 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4e9126f43d649ccae0f72de158b91d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e9126f43d649ccae0f72de158b91d9" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 41221 } }
I20260620 18:31:55.874769 10851 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4e9126f43d649ccae0f72de158b91d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:31:55.874964 10851 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4e9126f43d649ccae0f72de158b91d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:31:55.875150 10851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4e9126f43d649ccae0f72de158b91d9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:31:55.877429 10851 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4e9126f43d649ccae0f72de158b91d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e9126f43d649ccae0f72de158b91d9" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 41221 } }
I20260620 18:31:55.878000 10851 leader_election.cc:302] T 00000000000000000000000000000000 P c4e9126f43d649ccae0f72de158b91d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4e9126f43d649ccae0f72de158b91d9; no voters:
Jun 20 18:31:55 dist-test-slave-cbn3 krb5kdc[10771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:31:55.878733 10851 leader_election.cc:288] T 00000000000000000000000000000000 P c4e9126f43d649ccae0f72de158b91d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:31:55.878850 10856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4e9126f43d649ccae0f72de158b91d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:31:55.880373 10856 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4e9126f43d649ccae0f72de158b91d9 [term 1 LEADER]: Becoming Leader. State: Replica: c4e9126f43d649ccae0f72de158b91d9, State: Running, Role: LEADER
I20260620 18:31:55.880968 10856 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4e9126f43d649ccae0f72de158b91d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e9126f43d649ccae0f72de158b91d9" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 41221 } }
I20260620 18:31:55.881310 10851 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4e9126f43d649ccae0f72de158b91d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:31:55.886253 10858 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4e9126f43d649ccae0f72de158b91d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4e9126f43d649ccae0f72de158b91d9. Latest consensus state: current_term: 1 leader_uuid: "c4e9126f43d649ccae0f72de158b91d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e9126f43d649ccae0f72de158b91d9" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 41221 } } }
I20260620 18:31:55.886332 10857 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4e9126f43d649ccae0f72de158b91d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4e9126f43d649ccae0f72de158b91d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e9126f43d649ccae0f72de158b91d9" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 41221 } } }
I20260620 18:31:55.886684 10858 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4e9126f43d649ccae0f72de158b91d9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:31:55.886767 10857 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4e9126f43d649ccae0f72de158b91d9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:31:55.896561 10853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:31:55.854100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42330 (local address 127.10.131.62:41221)
0620 18:31:55.855086 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0620 18:31:55.855103 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 18:31:55.856337 (+ 1234us) server_negotiation.cc:407] Connection header received
0620 18:31:55.865309 (+ 8972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:31:55.865336 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:31:55.865825 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:31:55.866265 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:31:55.868893 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:55.870971 (+ 2078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:55.872556 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:55.873643 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:55.873876 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:31:55.880551 (+ 6675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:31:55.880562 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:31:55.880583 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:31:55.880642 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:31:55.886555 (+ 5913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:31:55.888018 (+ 1463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:31:55.888044 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:31:55.888056 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:31:55.888349 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:31:55.889128 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:31:55.889137 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:31:55.889144 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:31:55.890081 (+ 937us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:31:55.890462 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0620 18:31:55.893643 (+ 3181us) server_negotiation.cc:299] Negotiation successful
0620 18:31:55.894217 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":439,"threads_started":1}
I20260620 18:31:55.901180 10865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:31:55.905479 10865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:31:55.915369 10865 catalog_manager.cc:1347] Generated new cluster ID: 984160e3b9c64d5a811a5b041c8ad510
I20260620 18:31:55.915510 10865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:31:55.924628 10865 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:31:55.925935 10865 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:31:55.938740 10865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4e9126f43d649ccae0f72de158b91d9: Generated new TSK 0
I20260620 18:31:55.939559 10865 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.131.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab.
Entry for principal kudu/127.10.131.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab.
I20260620 18:31:55.996690 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.1:0
--local_ip_for_outbound_sockets=127.10.131.1
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:41221
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:31:56.342609 10877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:31:56.342935 10877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:31:56.343114 10877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:31:56.350373 10877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:31:56.350518 10877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:31:56.350703 10877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.1
I20260620 18:31:56.360284 10877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:41221
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.1
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:31:56.361569 10877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:31:56.362910 10877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:31:56.375955 10883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:31:56.375928 10885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:31:56.375998 10882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:31:56.382052 10877 server_base.cc:1034] running on GCE node
I20260620 18:31:56.382772 10877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:31:56.383747 10877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:31:56.384945 10877 hybrid_clock.cc:648] HybridClock initialized: now 1781980316384872 us; error 56 us; skew 500 ppm
I20260620 18:31:56.385273 10877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:31:56 dist-test-slave-cbn3 krb5kdc[10771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980316, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:31:56.388819 10877 init.cc:377] Logged in from keytab as kudu/127.10.131.1@KRBTEST.COM (short username kudu)
I20260620 18:31:56.390463 10877 webserver.cc:460] Webserver started at http://127.10.131.1:45745/ using document root <none> and password file <none>
I20260620 18:31:56.391235 10877 fs_manager.cc:359] Metadata directory not provided
I20260620 18:31:56.391382 10877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:31:56.391696 10877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:31:56.394914 10877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-0/data/instance:
uuid: "0dba68e29ff746908c54f9c317f5428b"
format_stamp: "Formatted at 2026-06-20 18:31:56 on dist-test-slave-cbn3"
server_key: "12c52e76327f8ace3e2d1ed90269565b"
server_key_iv: "287af2aeb0d0306b1843f3a0e5ef7d6c"
server_key_version: "encryptionkey@0"
I20260620 18:31:56.395738 10877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance:
uuid: "0dba68e29ff746908c54f9c317f5428b"
format_stamp: "Formatted at 2026-06-20 18:31:56 on dist-test-slave-cbn3"
server_key: "12c52e76327f8ace3e2d1ed90269565b"
server_key_iv: "287af2aeb0d0306b1843f3a0e5ef7d6c"
server_key_version: "encryptionkey@0"
I20260620 18:31:56.401000 10877 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 18:31:56.404341 10892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:31:56.405843 10877 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:31:56.406010 10877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-0/wal
uuid: "0dba68e29ff746908c54f9c317f5428b"
format_stamp: "Formatted at 2026-06-20 18:31:56 on dist-test-slave-cbn3"
server_key: "12c52e76327f8ace3e2d1ed90269565b"
server_key_iv: "287af2aeb0d0306b1843f3a0e5ef7d6c"
server_key_version: "encryptionkey@0"
I20260620 18:31:56.406244 10877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:31:56.420923 10877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:31:56.425230 10877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:31:56.425511 10877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:31:56.426440 10877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:31:56.427786 10877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:31:56.427901 10877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:31:56.428047 10877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:31:56.428124 10877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:31:56.462038 10877 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.1:35597
I20260620 18:31:56.462096 11004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.1:35597 every 8 connection(s)
I20260620 18:31:56.463357 10877 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
I20260620 18:31:56.465181 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 10877
I20260620 18:31:56.465442 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance
I20260620 18:31:56.466276 10764 external_mini_cluster.cc:1428] Setting key 38ef045c1855a0e4140734f328437c71
Jun 20 18:31:56 dist-test-slave-cbn3 krb5kdc[10771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980316, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.131.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.2@KRBTEST.COM" created.
I20260620 18:31:56.486253 11007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:31:56.466135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:43007 (local address 127.10.131.62:41221)
0620 18:31:56.466653 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0620 18:31:56.466657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:31:56.467932 (+ 1275us) server_negotiation.cc:407] Connection header received
0620 18:31:56.469960 (+ 2028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:31:56.469969 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:31:56.470065 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:31:56.470188 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:31:56.471681 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:56.472785 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:56.474722 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:56.475512 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:56.475655 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:31:56.479029 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:31:56.479035 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:31:56.479041 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:31:56.479089 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:31:56.482735 (+ 3646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:31:56.483552 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:31:56.483557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:31:56.483561 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:31:56.483680 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:31:56.484178 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:31:56.484183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:31:56.484186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:31:56.484450 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:31:56.484558 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 18:31:56.485529 (+ 971us) server_negotiation.cc:299] Negotiation successful
0620 18:31:56.485788 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":286,"threads_started":1}
I20260620 18:31:56.487923 11008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:31:56.466893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:41221 (local address 127.10.131.1:43007)
0620 18:31:56.467733 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0620 18:31:56.467809 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 18:31:56.469688 (+ 1879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:31:56.470395 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:31:56.470403 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:31:56.470943 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:31:56.471510 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:31:56.471527 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:31:56.472969 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:31:56.472977 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:31:56.474579 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:31:56.474592 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:31:56.475717 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:31:56.475721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:31:56.475915 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:31:56.476599 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:31:56.476621 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:31:56.478801 (+ 2180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:31:56.482884 (+ 4083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:31:56.482891 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:31:56.482903 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:31:56.483433 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:31:56.483811 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:31:56.483815 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:31:56.483818 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:31:56.484026 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:31:56.484571 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:31:56.484578 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:31:56.485057 (+ 479us) client_negotiation.cc:769] Sending connection context
0620 18:31:56.485391 (+ 334us) client_negotiation.cc:240] Negotiation successful
0620 18:31:56.486140 (+ 749us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":315,"threads_started":1}
I20260620 18:31:56.489774 11005 heartbeater.cc:347] Connected to a master server at 127.10.131.62:41221
I20260620 18:31:56.490080 11005 heartbeater.cc:464] Registering TS with master...
I20260620 18:31:56.491058 11005 heartbeater.cc:511] Master 127.10.131.62:41221 requested a full tablet report, sending...
I20260620 18:31:56.493456 10816 ts_manager.cc:194] Registered new tserver with Master: 0dba68e29ff746908c54f9c317f5428b (127.10.131.1:35597)
I20260620 18:31:56.494954 10816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.1@KRBTEST.COM'} at 127.10.131.1:43007
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab.
Entry for principal kudu/127.10.131.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab.
I20260620 18:31:56.509490 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.2:0
--local_ip_for_outbound_sockets=127.10.131.2
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:41221
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:31:56.827179 11011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:31:56.827455 11011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:31:56.827622 11011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:31:56.834718 11011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:31:56.834826 11011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:31:56.835000 11011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.2
I20260620 18:31:56.844581 11011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:41221
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.2
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:31:56.845822 11011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:31:56.847076 11011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:31:56.860227 11019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:31:56.860144 11016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:31:56.860283 11017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:31:56.861119 11011 server_base.cc:1034] running on GCE node
I20260620 18:31:56.861691 11011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:31:56.862591 11011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:31:56.863854 11011 hybrid_clock.cc:648] HybridClock initialized: now 1781980316863733 us; error 55 us; skew 500 ppm
I20260620 18:31:56.864192 11011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:31:56 dist-test-slave-cbn3 krb5kdc[10771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980316, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:31:56.867419 11011 init.cc:377] Logged in from keytab as kudu/127.10.131.2@KRBTEST.COM (short username kudu)
I20260620 18:31:56.868803 11011 webserver.cc:460] Webserver started at http://127.10.131.2:37201/ using document root <none> and password file <none>
I20260620 18:31:56.869496 11011 fs_manager.cc:359] Metadata directory not provided
I20260620 18:31:56.869598 11011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:31:56.869850 11011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:31:56.872747 11011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-1/data/instance:
uuid: "388f8428dd3d45129f81e50d1fcdd519"
format_stamp: "Formatted at 2026-06-20 18:31:56 on dist-test-slave-cbn3"
server_key: "49c9403e50ffce81300399599276fbd9"
server_key_iv: "b03a49905d2d8a05c88ca087de68290d"
server_key_version: "encryptionkey@0"
I20260620 18:31:56.873545 11011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance:
uuid: "388f8428dd3d45129f81e50d1fcdd519"
format_stamp: "Formatted at 2026-06-20 18:31:56 on dist-test-slave-cbn3"
server_key: "49c9403e50ffce81300399599276fbd9"
server_key_iv: "b03a49905d2d8a05c88ca087de68290d"
server_key_version: "encryptionkey@0"
I20260620 18:31:56.879127 11011 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:31:56.882628 11026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:31:56.883970 11011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:31:56.884166 11011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-1/wal
uuid: "388f8428dd3d45129f81e50d1fcdd519"
format_stamp: "Formatted at 2026-06-20 18:31:56 on dist-test-slave-cbn3"
server_key: "49c9403e50ffce81300399599276fbd9"
server_key_iv: "b03a49905d2d8a05c88ca087de68290d"
server_key_version: "encryptionkey@0"
I20260620 18:31:56.884418 11011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:31:56.908139 11011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:31:56.912956 11011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:31:56.913251 11011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:31:56.914283 11011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:31:56.915758 11011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:31:56.915875 11011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:31:56.916030 11011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:31:56.916111 11011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:31:56.949263 11011 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.2:40589
I20260620 18:31:56.949352 11138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.2:40589 every 8 connection(s)
I20260620 18:31:56.950551 11011 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
I20260620 18:31:56.955816 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 11011
I20260620 18:31:56.956019 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance
I20260620 18:31:56.956528 10764 external_mini_cluster.cc:1428] Setting key 63e36a147ad5e4ab1a29b373b85cd1f3
Jun 20 18:31:56 dist-test-slave-cbn3 krb5kdc[10771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980316, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.131.3@KRBTEST.COM; defaulting to no policy
I20260620 18:31:56.974341 11007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:31:56.953247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:60883 (local address 127.10.131.62:41221)
0620 18:31:56.953463 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 18:31:56.953470 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:31:56.954722 (+ 1252us) server_negotiation.cc:407] Connection header received
0620 18:31:56.957146 (+ 2424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:31:56.957171 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:31:56.957275 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:31:56.957397 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:31:56.959201 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:56.959918 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:56.962491 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:56.963046 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:56.963189 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:31:56.966967 (+ 3778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:31:56.966976 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:31:56.966984 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:31:56.967030 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:31:56.970809 (+ 3779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:31:56.971713 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:31:56.971719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:31:56.971724 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:31:56.971900 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:31:56.972336 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:31:56.972342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:31:56.972347 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:31:56.972738 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:31:56.972937 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 18:31:56.973680 (+ 743us) server_negotiation.cc:299] Negotiation successful
0620 18:31:56.973933 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 18:31:56.975095 11141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:31:56.953826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:41221 (local address 127.10.131.2:60883)
0620 18:31:56.954545 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0620 18:31:56.954587 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:31:56.956811 (+ 2224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:31:56.957690 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:31:56.957698 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:31:56.958233 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:31:56.959013 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:31:56.959038 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:31:56.960087 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:31:56.960095 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:31:56.962331 (+ 2236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:31:56.962349 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:31:56.963268 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:31:56.963273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:31:56.963465 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:31:56.964192 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:31:56.964214 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:31:56.966715 (+ 2501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:31:56.970995 (+ 4280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:31:56.971005 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:31:56.971025 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:31:56.971565 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:31:56.972018 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:31:56.972022 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:31:56.972024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:31:56.972208 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:31:56.972925 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:31:56.972932 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:31:56.973255 (+ 323us) client_negotiation.cc:769] Sending connection context
0620 18:31:56.973513 (+ 258us) client_negotiation.cc:240] Negotiation successful
0620 18:31:56.973908 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":296,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.3@KRBTEST.COM" created.
I20260620 18:31:56.976866 11139 heartbeater.cc:347] Connected to a master server at 127.10.131.62:41221
I20260620 18:31:56.977180 11139 heartbeater.cc:464] Registering TS with master...
I20260620 18:31:56.977972 11139 heartbeater.cc:511] Master 127.10.131.62:41221 requested a full tablet report, sending...
I20260620 18:31:56.979561 10816 ts_manager.cc:194] Registered new tserver with Master: 388f8428dd3d45129f81e50d1fcdd519 (127.10.131.2:40589)
I20260620 18:31:56.980773 10816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.2@KRBTEST.COM'} at 127.10.131.2:60883
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab.
Entry for principal kudu/127.10.131.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab.
I20260620 18:31:57.000641 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.3:0
--local_ip_for_outbound_sockets=127.10.131.3
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:41221
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:31:57.318158 11144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:31:57.318427 11144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:31:57.318600 11144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:31:57.325680 11144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:31:57.325783 11144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:31:57.325974 11144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.3
I20260620 18:31:57.335486 11144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:41221
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.3
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:31:57.336738 11144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:31:57.338094 11144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:31:57.350778 11152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:31:57.350829 11150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:31:57.350744 11149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:31:57.352038 11144 server_base.cc:1034] running on GCE node
I20260620 18:31:57.352574 11144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:31:57.353672 11144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:31:57.354928 11144 hybrid_clock.cc:648] HybridClock initialized: now 1781980317354894 us; error 62 us; skew 500 ppm
I20260620 18:31:57.355284 11144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:31:57 dist-test-slave-cbn3 krb5kdc[10771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980317, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:31:57.358857 11144 init.cc:377] Logged in from keytab as kudu/127.10.131.3@KRBTEST.COM (short username kudu)
I20260620 18:31:57.360621 11144 webserver.cc:460] Webserver started at http://127.10.131.3:38761/ using document root <none> and password file <none>
I20260620 18:31:57.361378 11144 fs_manager.cc:359] Metadata directory not provided
I20260620 18:31:57.361497 11144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:31:57.361768 11144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:31:57.364689 11144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-2/data/instance:
uuid: "34fad8a6459a49d7a5a861aa69d9bd34"
format_stamp: "Formatted at 2026-06-20 18:31:57 on dist-test-slave-cbn3"
server_key: "dc9e2c7a4032ef87c3369793a58f0319"
server_key_iv: "545130e582c89404643dc2fe4bba7e28"
server_key_version: "encryptionkey@0"
I20260620 18:31:57.365463 11144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance:
uuid: "34fad8a6459a49d7a5a861aa69d9bd34"
format_stamp: "Formatted at 2026-06-20 18:31:57 on dist-test-slave-cbn3"
server_key: "dc9e2c7a4032ef87c3369793a58f0319"
server_key_iv: "545130e582c89404643dc2fe4bba7e28"
server_key_version: "encryptionkey@0"
I20260620 18:31:57.370399 11144 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:31:57.373816 11159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:31:57.375092 11144 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:31:57.375211 11144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-2/wal
uuid: "34fad8a6459a49d7a5a861aa69d9bd34"
format_stamp: "Formatted at 2026-06-20 18:31:57 on dist-test-slave-cbn3"
server_key: "dc9e2c7a4032ef87c3369793a58f0319"
server_key_iv: "545130e582c89404643dc2fe4bba7e28"
server_key_version: "encryptionkey@0"
I20260620 18:31:57.375360 11144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:31:57.399643 11144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:31:57.403885 11144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:31:57.404125 11144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:31:57.405104 11144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:31:57.406406 11144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:31:57.406481 11144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:31:57.406566 11144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:31:57.406610 11144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:31:57.440768 11144 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.3:40889
I20260620 18:31:57.440908 11271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.3:40889 every 8 connection(s)
I20260620 18:31:57.442099 11144 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
I20260620 18:31:57.446650 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 11144
I20260620 18:31:57.446875 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance
I20260620 18:31:57.447369 10764 external_mini_cluster.cc:1428] Setting key f6b406506a18c5ade91cbdb98fa52933
Jun 20 18:31:57 dist-test-slave-cbn3 krb5kdc[10771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980317, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:31:57.463618 11007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:31:57.444924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:37579 (local address 127.10.131.62:41221)
0620 18:31:57.445157 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 18:31:57.445164 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:31:57.446444 (+ 1280us) server_negotiation.cc:407] Connection header received
0620 18:31:57.448567 (+ 2123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:31:57.448573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:31:57.448698 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:31:57.448807 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:31:57.450967 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:57.451841 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:57.453516 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:57.453918 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:57.454015 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:31:57.457491 (+ 3476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:31:57.457496 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:31:57.457499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:31:57.457532 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:31:57.460057 (+ 2525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:31:57.460717 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:31:57.460720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:31:57.460722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:31:57.460913 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:31:57.461364 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:31:57.461368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:31:57.461370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:31:57.461752 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:31:57.461896 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 18:31:57.462935 (+ 1039us) server_negotiation.cc:299] Negotiation successful
0620 18:31:57.463210 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 18:31:57.464533 11274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:31:57.445502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:41221 (local address 127.10.131.3:37579)
0620 18:31:57.446273 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0620 18:31:57.446318 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:31:57.448232 (+ 1914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:31:57.449154 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:31:57.449167 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:31:57.450006 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:31:57.450678 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:31:57.450707 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:31:57.451985 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:31:57.451988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:31:57.453386 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:31:57.453398 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:31:57.454111 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:31:57.454116 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:31:57.454307 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:31:57.454997 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:31:57.455029 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:31:57.457246 (+ 2217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:31:57.460182 (+ 2936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:31:57.460189 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:31:57.460202 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:31:57.460609 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:31:57.461008 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:31:57.461011 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:31:57.461013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:31:57.461253 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:31:57.461907 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:31:57.461914 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:31:57.462314 (+ 400us) client_negotiation.cc:769] Sending connection context
0620 18:31:57.462626 (+ 312us) client_negotiation.cc:240] Negotiation successful
0620 18:31:57.463182 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":341,"threads_started":1}
I20260620 18:31:57.466001 11272 heartbeater.cc:347] Connected to a master server at 127.10.131.62:41221
I20260620 18:31:57.466379 11272 heartbeater.cc:464] Registering TS with master...
I20260620 18:31:57.467150 11272 heartbeater.cc:511] Master 127.10.131.62:41221 requested a full tablet report, sending...
I20260620 18:31:57.468459 10816 ts_manager.cc:194] Registered new tserver with Master: 34fad8a6459a49d7a5a861aa69d9bd34 (127.10.131.3:40889)
I20260620 18:31:57.469280 10816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.3@KRBTEST.COM'} at 127.10.131.3:37579
I20260620 18:31:57.476341 10764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 18:31:57 dist-test-slave-cbn3 krb5kdc[10771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980317, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:31:57.498670 11005 heartbeater.cc:503] Master 127.10.131.62:41221 was elected leader, sending a full tablet report...
Password for test-user@KRBTEST.COM:
Jun 20 18:31:57 dist-test-slave-cbn3 krb5kdc[10771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980317, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.131.1@KRBTEST.COM
I20260620 18:31:57.522644 11279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:31:57.505981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36152 (local address 127.10.131.1:35597)
0620 18:31:57.506658 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0620 18:31:57.506667 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:31:57.506699 (+ 32us) server_negotiation.cc:407] Connection header received
0620 18:31:57.506892 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:31:57.506905 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:31:57.507124 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:31:57.507290 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:31:57.509082 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:57.510398 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:57.511711 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:57.512276 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:57.512433 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:31:57.516747 (+ 4314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:31:57.516758 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:31:57.516774 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:31:57.516829 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:31:57.519830 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:31:57.520528 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:31:57.520536 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:31:57.520538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:31:57.520659 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:31:57.521133 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:31:57.521137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:31:57.521139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:31:57.521498 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:31:57.521733 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 18:31:57.522038 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 18:31:57.522301 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":368,"threads_started":1}
W20260620 18:31:57.523929 10920 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36152
Jun 20 18:31:57 dist-test-slave-cbn3 krb5kdc[10771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 18:31:57 dist-test-slave-cbn3 krb5kdc[10771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.1@KRBTEST.COM
I20260620 18:31:57.567171 11279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:31:57.553410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36156 (local address 127.10.131.1:35597)
0620 18:31:57.553658 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 18:31:57.553666 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:31:57.553921 (+ 255us) server_negotiation.cc:407] Connection header received
0620 18:31:57.554143 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:31:57.554153 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:31:57.554216 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:31:57.554323 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:31:57.555629 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:57.556313 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:57.557622 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:57.558069 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:57.558207 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:31:57.562093 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:31:57.562098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:31:57.562102 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:31:57.562135 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:31:57.564591 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:31:57.565298 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:31:57.565306 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:31:57.565309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:31:57.565436 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:31:57.565828 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:31:57.565833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:31:57.565840 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:31:57.566078 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:31:57.566321 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 18:31:57.566636 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 18:31:57.566862 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 18:31:57.568586 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 10877
I20260620 18:31:57.581101 11279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:31:57.569646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36170 (local address 127.10.131.1:35597)
0620 18:31:57.569854 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 18:31:57.569860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:31:57.569992 (+ 132us) server_negotiation.cc:407] Connection header received
0620 18:31:57.570099 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:31:57.570103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:31:57.570155 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:31:57.570227 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:31:57.571348 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:57.572080 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:57.573459 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:57.573856 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:57.573967 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:31:57.576074 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:31:57.576079 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:31:57.576082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:31:57.576109 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:31:57.578776 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:31:57.579402 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:31:57.579407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:31:57.579411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:31:57.579540 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:31:57.579873 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:31:57.579876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:31:57.579877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:31:57.580102 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:31:57.580280 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 18:31:57.580605 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 18:31:57.580765 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:31:57.581825 11000 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:31:57.672668 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 10877
I20260620 18:31:57.689442 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 11011
Jun 20 18:31:57 dist-test-slave-cbn3 krb5kdc[10771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.2@KRBTEST.COM
I20260620 18:31:57.706202 11286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:31:57.690347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38150 (local address 127.10.131.2:40589)
0620 18:31:57.690903 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0620 18:31:57.690910 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:31:57.690935 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:31:57.690997 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:31:57.691004 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:31:57.691227 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:31:57.691398 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:31:57.692587 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:57.693701 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:57.695105 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:57.695664 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:57.695812 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:31:57.699824 (+ 4012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:31:57.699834 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:31:57.699849 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:31:57.699894 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:31:57.703187 (+ 3293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:31:57.703910 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:31:57.703924 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:31:57.703926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:31:57.704053 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:31:57.704496 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:31:57.704501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:31:57.704505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:31:57.704897 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:31:57.705229 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0620 18:31:57.705565 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 18:31:57.705841 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":283,"threads_started":1}
I20260620 18:31:57.707458 11134 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:31:57.790274 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 11011
I20260620 18:31:57.806500 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 11144
Jun 20 18:31:57 dist-test-slave-cbn3 krb5kdc[10771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.3@KRBTEST.COM
I20260620 18:31:57.823587 11288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:31:57.807691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46360 (local address 127.10.131.3:40889)
0620 18:31:57.808589 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0620 18:31:57.808596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:31:57.808625 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:31:57.808684 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:31:57.808691 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:31:57.808995 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:31:57.809134 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:31:57.810355 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:57.811401 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:57.813181 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:57.813756 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:57.813903 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:31:57.818053 (+ 4150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:31:57.818064 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:31:57.818077 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:31:57.818114 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:31:57.820733 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:31:57.821445 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:31:57.821453 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:31:57.821455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:31:57.821571 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:31:57.821976 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:31:57.821980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:31:57.821988 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:31:57.822309 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:31:57.822649 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0620 18:31:57.822959 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 18:31:57.823233 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":755,"thread_start_us":567,"threads_started":1}
I20260620 18:31:57.825115 11267 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:31:57.907771 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 11144
I20260620 18:31:57.923220 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 10784
I20260620 18:31:57.924453 10846 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:31:58.025259 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 10784
2026-06-20T18:31:58Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3895 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:31:58 dist-test-slave-cbn3 krb5kdc[11293](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:31:58 dist-test-slave-cbn3 krb5kdc[11293](info): set up 2 sockets
Jun 20 18:31:58 dist-test-slave-cbn3 krb5kdc[11293](info): commencing operation
krb5kdc: starting...
W20260620 18:31:59.150252 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:31:59 dist-test-slave-cbn3 krb5kdc[11293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:31:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:31:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.131.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
I20260620 18:31:59.319765 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:38863
--webserver_interface=127.10.131.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:38513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:38863
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:31:59.629614 11306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:31:59.629873 11306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:31:59.629956 11306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:31:59.637530 11306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:31:59.637634 11306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:31:59.637681 11306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:31:59.637737 11306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:31:59.637777 11306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:31:59.647241 11306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:38513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:38863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:38863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:31:59.648432 11306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:31:59.649902 11306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:31:59.661326 11312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:31:59.661463 11311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:31:59.661330 11314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:31:59.662386 11306 server_base.cc:1034] running on GCE node
I20260620 18:31:59.663120 11306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:31:59.664430 11306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:31:59.665674 11306 hybrid_clock.cc:648] HybridClock initialized: now 1781980319665627 us; error 62 us; skew 500 ppm
I20260620 18:31:59.666024 11306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:31:59 dist-test-slave-cbn3 krb5kdc[11293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980319, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:31:59.672035 11306 init.cc:377] Logged in from keytab as kudu/127.10.131.62@KRBTEST.COM (short username kudu)
I20260620 18:31:59.673674 11306 webserver.cc:460] Webserver started at http://127.10.131.62:39011/ using document root <none> and password file <none>
I20260620 18:31:59.674357 11306 fs_manager.cc:359] Metadata directory not provided
I20260620 18:31:59.674504 11306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:31:59.674824 11306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:31:59.677944 11306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "f3a20e0917184872a98066da1ed80392"
format_stamp: "Formatted at 2026-06-20 18:31:59 on dist-test-slave-cbn3"
server_key: "a66c71ffbcc6bd73ad774d8d34ba7d7f"
server_key_iv: "4006c9a7b6d0986f027067d94e8fcc2c"
server_key_version: "encryptionkey@0"
I20260620 18:31:59.678704 11306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "f3a20e0917184872a98066da1ed80392"
format_stamp: "Formatted at 2026-06-20 18:31:59 on dist-test-slave-cbn3"
server_key: "a66c71ffbcc6bd73ad774d8d34ba7d7f"
server_key_iv: "4006c9a7b6d0986f027067d94e8fcc2c"
server_key_version: "encryptionkey@0"
I20260620 18:31:59.683878 11306 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:31:59.687361 11321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:31:59.688647 11306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:31:59.688827 11306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "f3a20e0917184872a98066da1ed80392"
format_stamp: "Formatted at 2026-06-20 18:31:59 on dist-test-slave-cbn3"
server_key: "a66c71ffbcc6bd73ad774d8d34ba7d7f"
server_key_iv: "4006c9a7b6d0986f027067d94e8fcc2c"
server_key_version: "encryptionkey@0"
I20260620 18:31:59.689098 11306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:31:59.707161 11306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:31:59.711330 11306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:31:59.711642 11306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:31:59.729444 11306 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:38863
I20260620 18:31:59.729468 11372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:38863 every 8 connection(s)
I20260620 18:31:59.730764 11306 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:31:59.734393 11373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:31:59.734781 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 11306
I20260620 18:31:59.735020 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:31:59.735842 10764 external_mini_cluster.cc:1428] Setting key 8c465bd596ec9759875d67a71e905755
I20260620 18:31:59.748210 11373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3a20e0917184872a98066da1ed80392: Bootstrap starting.
Jun 20 18:31:59 dist-test-slave-cbn3 krb5kdc[11293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:31:59.751770 11373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3a20e0917184872a98066da1ed80392: Neither blocks nor log segments found. Creating new log.
I20260620 18:31:59.752960 11373 log.cc:824] T 00000000000000000000000000000000 P f3a20e0917184872a98066da1ed80392: Log is configured to *not* fsync() on all Append() calls
I20260620 18:31:59.755707 11373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3a20e0917184872a98066da1ed80392: No bootstrap required, opened a new log
I20260620 18:31:59.759233 11376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:31:59.737907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50046 (local address 127.10.131.62:38863)
0620 18:31:59.738689 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0620 18:31:59.738700 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:31:59.738740 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:31:59.740651 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:31:59.740673 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:31:59.741182 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:31:59.741502 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:31:59.743172 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:59.744426 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:59.746198 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:31:59.746858 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:31:59.746982 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:31:59.751496 (+ 4514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:31:59.751503 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:31:59.751517 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:31:59.751550 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:31:59.754673 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:31:59.755356 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:31:59.755362 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:31:59.755368 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:31:59.755519 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:31:59.755938 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:31:59.755943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:31:59.755946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:31:59.756635 (+ 689us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:31:59.756952 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0620 18:31:59.757393 (+ 441us) server_negotiation.cc:299] Negotiation successful
0620 18:31:59.757861 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":349,"threads_started":1}
I20260620 18:31:59.764101 11373 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3a20e0917184872a98066da1ed80392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3a20e0917184872a98066da1ed80392" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38863 } }
I20260620 18:31:59.764432 11373 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3a20e0917184872a98066da1ed80392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:31:59.764503 11373 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3a20e0917184872a98066da1ed80392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3a20e0917184872a98066da1ed80392, State: Initialized, Role: FOLLOWER
I20260620 18:31:59.765029 11373 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3a20e0917184872a98066da1ed80392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3a20e0917184872a98066da1ed80392" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38863 } }
I20260620 18:31:59.765231 11373 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3a20e0917184872a98066da1ed80392 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:31:59.765326 11373 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3a20e0917184872a98066da1ed80392 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:31:59.765470 11373 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3a20e0917184872a98066da1ed80392 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:31:59.767586 11373 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3a20e0917184872a98066da1ed80392 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3a20e0917184872a98066da1ed80392" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38863 } }
I20260620 18:31:59.768040 11373 leader_election.cc:302] T 00000000000000000000000000000000 P f3a20e0917184872a98066da1ed80392 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3a20e0917184872a98066da1ed80392; no voters:
I20260620 18:31:59.768589 11373 leader_election.cc:288] T 00000000000000000000000000000000 P f3a20e0917184872a98066da1ed80392 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:31:59.768751 11378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3a20e0917184872a98066da1ed80392 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:31:59.769423 11378 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3a20e0917184872a98066da1ed80392 [term 1 LEADER]: Becoming Leader. State: Replica: f3a20e0917184872a98066da1ed80392, State: Running, Role: LEADER
I20260620 18:31:59.769896 11378 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3a20e0917184872a98066da1ed80392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3a20e0917184872a98066da1ed80392" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38863 } }
I20260620 18:31:59.770545 11373 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3a20e0917184872a98066da1ed80392 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:31:59.772483 11380 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3a20e0917184872a98066da1ed80392 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3a20e0917184872a98066da1ed80392. Latest consensus state: current_term: 1 leader_uuid: "f3a20e0917184872a98066da1ed80392" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3a20e0917184872a98066da1ed80392" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38863 } } }
I20260620 18:31:59.772833 11380 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3a20e0917184872a98066da1ed80392 [sys.catalog]: This master's current role is: LEADER
I20260620 18:31:59.774585 11379 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3a20e0917184872a98066da1ed80392 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3a20e0917184872a98066da1ed80392" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3a20e0917184872a98066da1ed80392" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38863 } } }
I20260620 18:31:59.775214 11379 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3a20e0917184872a98066da1ed80392 [sys.catalog]: This master's current role is: LEADER
I20260620 18:31:59.776084 11385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:31:59.782015 11385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:31:59.790725 11385 catalog_manager.cc:1347] Generated new cluster ID: 9a59786da3134a55961ab6581fba5229
I20260620 18:31:59.790824 11385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:31:59.815826 11385 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:31:59.817226 11385 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:31:59.831856 11385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3a20e0917184872a98066da1ed80392: Generated new TSK 0
I20260620 18:31:59.832696 11385 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.131.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab.
Entry for principal kudu/127.10.131.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab.
I20260620 18:31:59.882630 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.1:0
--local_ip_for_outbound_sockets=127.10.131.1
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:38863
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:38513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:00.205842 11399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:00.206131 11399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:00.206314 11399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:00.213578 11399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:00.213718 11399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:00.213909 11399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.1
I20260620 18:32:00.223758 11399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:38513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:38863
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.1
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:00.225096 11399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:00.226411 11399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:00.239730 11407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:00.239759 11405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:00.239759 11404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:00.240592 11399 server_base.cc:1034] running on GCE node
I20260620 18:32:00.241238 11399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:00.242168 11399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:00.243378 11399 hybrid_clock.cc:648] HybridClock initialized: now 1781980320243340 us; error 38 us; skew 500 ppm
I20260620 18:32:00.243690 11399 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:00 dist-test-slave-cbn3 krb5kdc[11293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980320, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:00.247004 11399 init.cc:377] Logged in from keytab as kudu/127.10.131.1@KRBTEST.COM (short username kudu)
I20260620 18:32:00.249576 11399 webserver.cc:460] Webserver started at http://127.10.131.1:41343/ using document root <none> and password file <none>
I20260620 18:32:00.250274 11399 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:00.250412 11399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:00.250742 11399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:00.253675 11399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-0/data/instance:
uuid: "e7228cda787344cda42bcd27a6c53eb3"
format_stamp: "Formatted at 2026-06-20 18:32:00 on dist-test-slave-cbn3"
server_key: "f1662adca0de39abf6f95e55a183526d"
server_key_iv: "e2432eba3ea3fa3886af8719eca8f767"
server_key_version: "encryptionkey@0"
I20260620 18:32:00.254556 11399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance:
uuid: "e7228cda787344cda42bcd27a6c53eb3"
format_stamp: "Formatted at 2026-06-20 18:32:00 on dist-test-slave-cbn3"
server_key: "f1662adca0de39abf6f95e55a183526d"
server_key_iv: "e2432eba3ea3fa3886af8719eca8f767"
server_key_version: "encryptionkey@0"
I20260620 18:32:00.259630 11399 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:32:00.263046 11414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:00.264349 11399 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 18:32:00.264518 11399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-0/wal
uuid: "e7228cda787344cda42bcd27a6c53eb3"
format_stamp: "Formatted at 2026-06-20 18:32:00 on dist-test-slave-cbn3"
server_key: "f1662adca0de39abf6f95e55a183526d"
server_key_iv: "e2432eba3ea3fa3886af8719eca8f767"
server_key_version: "encryptionkey@0"
I20260620 18:32:00.264734 11399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:00.287814 11399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:00.292510 11399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:00.292886 11399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:00.293920 11399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:00.295212 11399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:00.295282 11399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:00.295457 11399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:00.295521 11399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:00.328222 11399 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.1:33967
I20260620 18:32:00.328310 11526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.1:33967 every 8 connection(s)
I20260620 18:32:00.329517 11399 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
I20260620 18:32:00.330693 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 11399
I20260620 18:32:00.330919 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance
I20260620 18:32:00.331647 10764 external_mini_cluster.cc:1428] Setting key db4c00f68af41381dcd3747f8ba97847
Jun 20 18:32:00 dist-test-slave-cbn3 krb5kdc[11293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980320, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.131.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.2@KRBTEST.COM" created.
I20260620 18:32:00.351083 11530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:00.333527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:39515 (local address 127.10.131.62:38863)
0620 18:32:00.334417 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:00.334424 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:00.334453 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:32:00.335997 (+ 1544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:00.336005 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:00.336094 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:00.336218 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:00.337825 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:00.338552 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:00.340263 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:00.340668 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:00.340807 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:00.344491 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:00.344500 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:00.344506 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:00.344555 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:00.348165 (+ 3610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:00.348955 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:00.348962 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:00.348969 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:00.349092 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:00.349480 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:00.349484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:00.349486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:00.349756 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:00.349866 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:00.350599 (+ 733us) server_negotiation.cc:299] Negotiation successful
0620 18:32:00.350829 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":781,"thread_start_us":3365,"threads_started":1}
I20260620 18:32:00.352576 11529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:00.332950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38863 (local address 127.10.131.1:39515)
0620 18:32:00.334062 (+ 1112us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:00.334102 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:00.335729 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:00.336455 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:00.336463 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:00.337064 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:00.337623 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:00.337641 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:00.338766 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:00.338774 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:00.340110 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:00.340122 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:00.340919 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:00.340927 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:00.341156 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:00.342030 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:00.342063 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:00.344248 (+ 2185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:00.348313 (+ 4065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:00.348320 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:00.348333 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:00.348762 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:00.349195 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:00.349199 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:00.349201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:00.349378 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:00.349851 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:00.349857 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:00.350188 (+ 331us) client_negotiation.cc:769] Sending connection context
0620 18:32:00.350437 (+ 249us) client_negotiation.cc:240] Negotiation successful
0620 18:32:00.351009 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":309,"threads_started":1}
I20260620 18:32:00.354233 11527 heartbeater.cc:347] Connected to a master server at 127.10.131.62:38863
I20260620 18:32:00.354566 11527 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:00.355324 11527 heartbeater.cc:511] Master 127.10.131.62:38863 requested a full tablet report, sending...
I20260620 18:32:00.357576 11338 ts_manager.cc:194] Registered new tserver with Master: e7228cda787344cda42bcd27a6c53eb3 (127.10.131.1:33967)
I20260620 18:32:00.359030 11338 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.1@KRBTEST.COM'} at 127.10.131.1:39515
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab.
Entry for principal kudu/127.10.131.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab.
I20260620 18:32:00.372596 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.2:0
--local_ip_for_outbound_sockets=127.10.131.2
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:38863
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:38513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:00.685099 11533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:00.685413 11533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:00.685585 11533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:00.692646 11533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:00.692792 11533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:00.693005 11533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.2
I20260620 18:32:00.702647 11533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:38513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:38863
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.2
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:00.703958 11533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:00.705331 11533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:00.718310 11541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:00.718398 11539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:00.718398 11538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:00.719200 11533 server_base.cc:1034] running on GCE node
I20260620 18:32:00.719770 11533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:00.720690 11533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:00.721908 11533 hybrid_clock.cc:648] HybridClock initialized: now 1781980320721836 us; error 61 us; skew 500 ppm
I20260620 18:32:00.722227 11533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:00 dist-test-slave-cbn3 krb5kdc[11293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980320, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:00.725214 11533 init.cc:377] Logged in from keytab as kudu/127.10.131.2@KRBTEST.COM (short username kudu)
I20260620 18:32:00.726605 11533 webserver.cc:460] Webserver started at http://127.10.131.2:36001/ using document root <none> and password file <none>
I20260620 18:32:00.727272 11533 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:00.727411 11533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:00.727720 11533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:00.730568 11533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-1/data/instance:
uuid: "392cecf516d34c09b7c59f0e7f6fbf46"
format_stamp: "Formatted at 2026-06-20 18:32:00 on dist-test-slave-cbn3"
server_key: "8bd8c5c885c7df245d998631a8d002b1"
server_key_iv: "91684d796d54af58a49b2b37358d735e"
server_key_version: "encryptionkey@0"
I20260620 18:32:00.731330 11533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance:
uuid: "392cecf516d34c09b7c59f0e7f6fbf46"
format_stamp: "Formatted at 2026-06-20 18:32:00 on dist-test-slave-cbn3"
server_key: "8bd8c5c885c7df245d998631a8d002b1"
server_key_iv: "91684d796d54af58a49b2b37358d735e"
server_key_version: "encryptionkey@0"
I20260620 18:32:00.736399 11533 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:32:00.739965 11548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:00.741519 11533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 18:32:00.741698 11533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-1/wal
uuid: "392cecf516d34c09b7c59f0e7f6fbf46"
format_stamp: "Formatted at 2026-06-20 18:32:00 on dist-test-slave-cbn3"
server_key: "8bd8c5c885c7df245d998631a8d002b1"
server_key_iv: "91684d796d54af58a49b2b37358d735e"
server_key_version: "encryptionkey@0"
I20260620 18:32:00.741950 11533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:00.767524 11533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:00.771940 11533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:00.772243 11533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:00.773314 11533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:00.774644 11533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:00.774765 11533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:00.774906 11533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:00.774979 11533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:00.808094 11533 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.2:44163
I20260620 18:32:00.808144 11660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.2:44163 every 8 connection(s)
I20260620 18:32:00.809413 11533 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
I20260620 18:32:00.818382 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 11533
I20260620 18:32:00.818572 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance
I20260620 18:32:00.819109 10764 external_mini_cluster.cc:1428] Setting key a1f2efe2afedf50e77b3ac1b82fa289b
Jun 20 18:32:00 dist-test-slave-cbn3 krb5kdc[11293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980320, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:32:00.830041 11530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:00.812012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:59977 (local address 127.10.131.62:38863)
0620 18:32:00.812181 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:00.812185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:00.813259 (+ 1074us) server_negotiation.cc:407] Connection header received
0620 18:32:00.815091 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:00.815095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:00.815164 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:00.815254 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:00.816658 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:00.817452 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:00.819745 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:00.820418 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:00.820558 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:00.824297 (+ 3739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:00.824302 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:00.824305 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:00.824336 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:00.826940 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:00.827648 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:00.827655 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:00.827658 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:00.827819 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:00.828250 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:00.828257 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:00.828260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:00.828573 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:00.828739 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:00.829460 (+ 721us) server_negotiation.cc:299] Negotiation successful
0620 18:32:00.829679 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:32:00.831144 11663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:00.812374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38863 (local address 127.10.131.2:59977)
0620 18:32:00.813105 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:00.813143 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:00.814796 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:00.815450 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:00.815458 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:00.815978 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:00.816502 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:00.816517 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:00.817585 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:00.817588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:00.819597 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:00.819612 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:00.820646 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:00.820651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:00.820975 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:00.821659 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:00.821680 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:00.824061 (+ 2381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:00.827085 (+ 3024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:00.827093 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:00.827109 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:00.827521 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:00.827938 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:00.827943 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:00.827947 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:00.828137 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:00.828716 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:00.828724 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:00.829116 (+ 392us) client_negotiation.cc:769] Sending connection context
0620 18:32:00.829387 (+ 271us) client_negotiation.cc:240] Negotiation successful
0620 18:32:00.829894 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":254,"threads_started":1}
I20260620 18:32:00.832790 11661 heartbeater.cc:347] Connected to a master server at 127.10.131.62:38863
I20260620 18:32:00.833134 11661 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:00.833841 11661 heartbeater.cc:511] Master 127.10.131.62:38863 requested a full tablet report, sending...
I20260620 18:32:00.835582 11338 ts_manager.cc:194] Registered new tserver with Master: 392cecf516d34c09b7c59f0e7f6fbf46 (127.10.131.2:44163)
WARNING: no policy specified for kudu/127.10.131.3@KRBTEST.COM; defaulting to no policy
I20260620 18:32:00.836506 11338 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.2@KRBTEST.COM'} at 127.10.131.2:59977
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab.
Entry for principal kudu/127.10.131.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab.
I20260620 18:32:00.860541 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.3:0
--local_ip_for_outbound_sockets=127.10.131.3
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:38863
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:38513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:01.193490 11666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:01.193820 11666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:01.193994 11666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:01.201285 11666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:01.201437 11666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:01.201643 11666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.3
I20260620 18:32:01.211274 11666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:38513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:38863
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.3
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:01.212643 11666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:01.214080 11666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:01.227378 11671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:01.227427 11672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:01.227451 11674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:01.228266 11666 server_base.cc:1034] running on GCE node
I20260620 18:32:01.229031 11666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:01.229961 11666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:01.231151 11666 hybrid_clock.cc:648] HybridClock initialized: now 1781980321231087 us; error 57 us; skew 500 ppm
I20260620 18:32:01.231446 11666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:01 dist-test-slave-cbn3 krb5kdc[11293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980321, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:01.234707 11666 init.cc:377] Logged in from keytab as kudu/127.10.131.3@KRBTEST.COM (short username kudu)
I20260620 18:32:01.236325 11666 webserver.cc:460] Webserver started at http://127.10.131.3:41373/ using document root <none> and password file <none>
I20260620 18:32:01.237062 11666 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:01.237165 11666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:01.237412 11666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:01.240229 11666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-2/data/instance:
uuid: "b468ca447ffb4af4b469d8765b1c5c18"
format_stamp: "Formatted at 2026-06-20 18:32:01 on dist-test-slave-cbn3"
server_key: "6d3bc31b4c72355e719ddb1ff74001ca"
server_key_iv: "f6d7dc757aa5d1cef2ac94c075d9254a"
server_key_version: "encryptionkey@0"
I20260620 18:32:01.241016 11666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance:
uuid: "b468ca447ffb4af4b469d8765b1c5c18"
format_stamp: "Formatted at 2026-06-20 18:32:01 on dist-test-slave-cbn3"
server_key: "6d3bc31b4c72355e719ddb1ff74001ca"
server_key_iv: "f6d7dc757aa5d1cef2ac94c075d9254a"
server_key_version: "encryptionkey@0"
I20260620 18:32:01.245909 11666 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:32:01.249385 11681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:01.251127 11666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 18:32:01.251286 11666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-2/wal
uuid: "b468ca447ffb4af4b469d8765b1c5c18"
format_stamp: "Formatted at 2026-06-20 18:32:01 on dist-test-slave-cbn3"
server_key: "6d3bc31b4c72355e719ddb1ff74001ca"
server_key_iv: "f6d7dc757aa5d1cef2ac94c075d9254a"
server_key_version: "encryptionkey@0"
I20260620 18:32:01.251540 11666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:01.271984 11666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:01.276448 11666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:01.276813 11666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:01.277894 11666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:01.279186 11666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:01.279294 11666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:01.279457 11666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:01.279531 11666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:01.313470 11666 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.3:40035
I20260620 18:32:01.313503 11793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.3:40035 every 8 connection(s)
I20260620 18:32:01.314764 11666 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
I20260620 18:32:01.321923 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 11666
I20260620 18:32:01.322070 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance
I20260620 18:32:01.322500 10764 external_mini_cluster.cc:1428] Setting key 4711e93166581f745bb7f135dd6a2be0
Jun 20 18:32:01 dist-test-slave-cbn3 krb5kdc[11293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980321, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:32:01.335448 11530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:01.317415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:42585 (local address 127.10.131.62:38863)
0620 18:32:01.317585 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:01.317589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:01.318825 (+ 1236us) server_negotiation.cc:407] Connection header received
0620 18:32:01.320726 (+ 1901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:01.320732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:01.320817 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:01.320950 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:01.322547 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:01.323659 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:01.325801 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:01.326184 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:01.326277 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:01.329535 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:01.329542 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:01.329547 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:01.329583 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:01.332944 (+ 3361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:01.333600 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:01.333604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:01.333606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:01.333717 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:01.334090 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:01.334093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:01.334095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:01.334340 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:01.334444 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:01.335055 (+ 611us) server_negotiation.cc:299] Negotiation successful
0620 18:32:01.335213 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:32:01.336472 11796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:01.317938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38863 (local address 127.10.131.3:42585)
0620 18:32:01.318649 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:01.318694 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:01.320405 (+ 1711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:01.321165 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:01.321173 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:01.321647 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:01.322381 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:01.322400 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:01.323853 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:01.323858 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:01.325677 (+ 1819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:01.325689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:01.326365 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:01.326370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:01.326547 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:01.327135 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:01.327160 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:01.329339 (+ 2179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:01.333073 (+ 3734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:01.333079 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:01.333090 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:01.333481 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:01.333809 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:01.333812 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:01.333814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:01.333999 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:01.334439 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:01.334445 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:01.334733 (+ 288us) client_negotiation.cc:769] Sending connection context
0620 18:32:01.334981 (+ 248us) client_negotiation.cc:240] Negotiation successful
0620 18:32:01.335311 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":286,"threads_started":1}
I20260620 18:32:01.338038 11794 heartbeater.cc:347] Connected to a master server at 127.10.131.62:38863
I20260620 18:32:01.338325 11794 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:01.338997 11794 heartbeater.cc:511] Master 127.10.131.62:38863 requested a full tablet report, sending...
I20260620 18:32:01.340319 11337 ts_manager.cc:194] Registered new tserver with Master: b468ca447ffb4af4b469d8765b1c5c18 (127.10.131.3:40035)
I20260620 18:32:01.341224 11337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.3@KRBTEST.COM'} at 127.10.131.3:42585
I20260620 18:32:01.349920 10764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:32:01.362145 11527 heartbeater.cc:503] Master 127.10.131.62:38863 was elected leader, sending a full tablet report...
I20260620 18:32:01.368640 11530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:01.356200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50058 (local address 127.10.131.62:38863)
0620 18:32:01.356426 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:01.356434 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:01.356962 (+ 528us) server_negotiation.cc:407] Connection header received
0620 18:32:01.357109 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:01.357114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:01.357175 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:01.357254 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:01.358623 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:01.359324 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:01.360549 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:01.360999 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:01.361096 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:01.363541 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:01.363546 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:01.363550 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:01.363580 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:01.366014 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:01.366663 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:01.366670 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:01.366673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:01.366808 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:01.367217 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:01.367224 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:01.367230 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:01.367538 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:01.367812 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:01.368108 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 18:32:01.368322 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 18:32:01.375059 11337 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50058:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:32:01.381942 11337 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:32:01.405525 11809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:01.395005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37308 (local address 127.10.131.3:40035)
0620 18:32:01.396169 (+ 1164us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:01.396183 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:01.397984 (+ 1801us) server_negotiation.cc:407] Connection header received
0620 18:32:01.398211 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:01.398219 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:01.398446 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:01.398631 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:01.399272 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:01.400908 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:01.402987 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:01.404325 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:01.404469 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:01.404702 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:01.405032 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 18:32:01.405192 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":888,"thread_start_us":672,"threads_started":1}
I20260620 18:32:01.405619 11808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:01.394463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:40035 (local address 127.0.0.1:37308)
0620 18:32:01.397847 (+ 3384us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:01.397865 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:01.398018 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:01.398621 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:01.398625 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:01.398649 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:01.399039 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:01.399055 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:01.401049 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:01.401063 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:01.402819 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:01.402831 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:01.404524 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:01.404531 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:01.404705 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:01.404771 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 18:32:01.404969 (+ 198us) client_negotiation.cc:240] Negotiation successful
0620 18:32:01.405156 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3250,"mutex_wait_us":25,"thread_start_us":4858,"threads_started":1}
I20260620 18:32:01.406117 11806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:01.394110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:44163 (local address 127.0.0.1:40704)
0620 18:32:01.397056 (+ 2946us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:01.397095 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:01.397297 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:01.398121 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:01.398130 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:01.398168 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:01.398551 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:01.398565 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:01.400979 (+ 2414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:01.400984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:01.403349 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:01.403364 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:01.405231 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:01.405237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:01.405410 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:01.405460 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 18:32:01.405530 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 18:32:01.405681 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2681,"mutex_wait_us":2,"thread_start_us":2555,"threads_started":1}
I20260620 18:32:01.407171 11807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:01.394312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40704 (local address 127.10.131.2:44163)
0620 18:32:01.396653 (+ 2341us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:01.396662 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:01.397208 (+ 546us) server_negotiation.cc:407] Connection header received
0620 18:32:01.397527 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:01.397562 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:01.397840 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:01.398107 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:01.398755 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:01.400762 (+ 2007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:01.403517 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:01.404996 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:01.406460 (+ 1464us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:01.406641 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:01.406704 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 18:32:01.406860 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2177,"thread_start_us":2032,"threads_started":1}
I20260620 18:32:01.408643 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:01.397977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:33967 (local address 127.0.0.1:58410)
0620 18:32:01.399953 (+ 1976us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:01.399980 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:01.400159 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:01.400951 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:01.400958 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:01.400998 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:01.401534 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:01.401547 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:01.403442 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:01.403447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:01.405595 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:01.405760 (+ 165us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:01.407852 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:01.407858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:01.408037 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:01.408065 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 18:32:01.408148 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 18:32:01.408279 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1858,"mutex_wait_us":1,"thread_start_us":1721,"threads_started":1}
I20260620 18:32:01.409395 11811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:01.398650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58410 (local address 127.10.131.1:33967)
0620 18:32:01.399465 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:01.399474 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:01.400074 (+ 600us) server_negotiation.cc:407] Connection header received
0620 18:32:01.400315 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:01.400324 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:01.400517 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:01.400657 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:01.401724 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:01.403273 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:01.405929 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:01.407355 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:01.407513 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:01.407706 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:01.408913 (+ 1207us) server_negotiation.cc:299] Negotiation successful
0620 18:32:01.409085 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":494,"threads_started":1}
I20260620 18:32:01.409693 11729 tablet_service.cc:1467] Processing CreateTablet for tablet 7674db37693b4e63976fcd1cb332d74a (DEFAULT_TABLE table=test-table [id=256f12806c0f454b8f77eea97b8c8746]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:01.410478 11596 tablet_service.cc:1467] Processing CreateTablet for tablet 7674db37693b4e63976fcd1cb332d74a (DEFAULT_TABLE table=test-table [id=256f12806c0f454b8f77eea97b8c8746]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:01.411012 11729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7674db37693b4e63976fcd1cb332d74a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:01.411803 11596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7674db37693b4e63976fcd1cb332d74a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:01.412765 11462 tablet_service.cc:1467] Processing CreateTablet for tablet 7674db37693b4e63976fcd1cb332d74a (DEFAULT_TABLE table=test-table [id=256f12806c0f454b8f77eea97b8c8746]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:01.414119 11462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7674db37693b4e63976fcd1cb332d74a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:01.422348 11812 tablet_bootstrap.cc:492] T 7674db37693b4e63976fcd1cb332d74a P b468ca447ffb4af4b469d8765b1c5c18: Bootstrap starting.
I20260620 18:32:01.425650 11812 tablet_bootstrap.cc:654] T 7674db37693b4e63976fcd1cb332d74a P b468ca447ffb4af4b469d8765b1c5c18: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:01.426843 11812 log.cc:824] T 7674db37693b4e63976fcd1cb332d74a P b468ca447ffb4af4b469d8765b1c5c18: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:01.429200 11815 tablet_bootstrap.cc:492] T 7674db37693b4e63976fcd1cb332d74a P e7228cda787344cda42bcd27a6c53eb3: Bootstrap starting.
I20260620 18:32:01.429842 11812 tablet_bootstrap.cc:492] T 7674db37693b4e63976fcd1cb332d74a P b468ca447ffb4af4b469d8765b1c5c18: No bootstrap required, opened a new log
I20260620 18:32:01.430104 11812 ts_tablet_manager.cc:1397] T 7674db37693b4e63976fcd1cb332d74a P b468ca447ffb4af4b469d8765b1c5c18: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 18:32:01.431978 11815 tablet_bootstrap.cc:654] T 7674db37693b4e63976fcd1cb332d74a P e7228cda787344cda42bcd27a6c53eb3: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:01.433001 11813 tablet_bootstrap.cc:492] T 7674db37693b4e63976fcd1cb332d74a P 392cecf516d34c09b7c59f0e7f6fbf46: Bootstrap starting.
I20260620 18:32:01.433040 11815 log.cc:824] T 7674db37693b4e63976fcd1cb332d74a P e7228cda787344cda42bcd27a6c53eb3: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:01.436141 11815 tablet_bootstrap.cc:492] T 7674db37693b4e63976fcd1cb332d74a P e7228cda787344cda42bcd27a6c53eb3: No bootstrap required, opened a new log
I20260620 18:32:01.436409 11815 ts_tablet_manager.cc:1397] T 7674db37693b4e63976fcd1cb332d74a P e7228cda787344cda42bcd27a6c53eb3: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 18:32:01.436419 11813 tablet_bootstrap.cc:654] T 7674db37693b4e63976fcd1cb332d74a P 392cecf516d34c09b7c59f0e7f6fbf46: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:01.437613 11813 log.cc:824] T 7674db37693b4e63976fcd1cb332d74a P 392cecf516d34c09b7c59f0e7f6fbf46: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:01.438047 11812 raft_consensus.cc:348] T 7674db37693b4e63976fcd1cb332d74a P b468ca447ffb4af4b469d8765b1c5c18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392cecf516d34c09b7c59f0e7f6fbf46" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 44163 } } peers { permanent_uuid: "b468ca447ffb4af4b469d8765b1c5c18" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40035 } } peers { permanent_uuid: "e7228cda787344cda42bcd27a6c53eb3" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 33967 } }
I20260620 18:32:01.438516 11812 raft_consensus.cc:374] T 7674db37693b4e63976fcd1cb332d74a P b468ca447ffb4af4b469d8765b1c5c18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:01.438628 11812 raft_consensus.cc:729] T 7674db37693b4e63976fcd1cb332d74a P b468ca447ffb4af4b469d8765b1c5c18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b468ca447ffb4af4b469d8765b1c5c18, State: Initialized, Role: FOLLOWER
I20260620 18:32:01.439186 11812 consensus_queue.cc:260] T 7674db37693b4e63976fcd1cb332d74a P b468ca447ffb4af4b469d8765b1c5c18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392cecf516d34c09b7c59f0e7f6fbf46" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 44163 } } peers { permanent_uuid: "b468ca447ffb4af4b469d8765b1c5c18" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40035 } } peers { permanent_uuid: "e7228cda787344cda42bcd27a6c53eb3" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 33967 } }
I20260620 18:32:01.440196 11794 heartbeater.cc:503] Master 127.10.131.62:38863 was elected leader, sending a full tablet report...
I20260620 18:32:01.440621 11813 tablet_bootstrap.cc:492] T 7674db37693b4e63976fcd1cb332d74a P 392cecf516d34c09b7c59f0e7f6fbf46: No bootstrap required, opened a new log
I20260620 18:32:01.440901 11812 ts_tablet_manager.cc:1428] T 7674db37693b4e63976fcd1cb332d74a P b468ca447ffb4af4b469d8765b1c5c18: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260620 18:32:01.440912 11813 ts_tablet_manager.cc:1397] T 7674db37693b4e63976fcd1cb332d74a P 392cecf516d34c09b7c59f0e7f6fbf46: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20260620 18:32:01.446938 11815 raft_consensus.cc:348] T 7674db37693b4e63976fcd1cb332d74a P e7228cda787344cda42bcd27a6c53eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392cecf516d34c09b7c59f0e7f6fbf46" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 44163 } } peers { permanent_uuid: "b468ca447ffb4af4b469d8765b1c5c18" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40035 } } peers { permanent_uuid: "e7228cda787344cda42bcd27a6c53eb3" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 33967 } }
I20260620 18:32:01.447418 11815 raft_consensus.cc:374] T 7674db37693b4e63976fcd1cb332d74a P e7228cda787344cda42bcd27a6c53eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:01.447523 11815 raft_consensus.cc:729] T 7674db37693b4e63976fcd1cb332d74a P e7228cda787344cda42bcd27a6c53eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7228cda787344cda42bcd27a6c53eb3, State: Initialized, Role: FOLLOWER
I20260620 18:32:01.448154 11815 consensus_queue.cc:260] T 7674db37693b4e63976fcd1cb332d74a P e7228cda787344cda42bcd27a6c53eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392cecf516d34c09b7c59f0e7f6fbf46" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 44163 } } peers { permanent_uuid: "b468ca447ffb4af4b469d8765b1c5c18" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40035 } } peers { permanent_uuid: "e7228cda787344cda42bcd27a6c53eb3" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 33967 } }
I20260620 18:32:01.450001 11815 ts_tablet_manager.cc:1428] T 7674db37693b4e63976fcd1cb332d74a P e7228cda787344cda42bcd27a6c53eb3: Time spent starting tablet: real 0.013s user 0.007s sys 0.008s
I20260620 18:32:01.451303 11813 raft_consensus.cc:348] T 7674db37693b4e63976fcd1cb332d74a P 392cecf516d34c09b7c59f0e7f6fbf46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392cecf516d34c09b7c59f0e7f6fbf46" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 44163 } } peers { permanent_uuid: "b468ca447ffb4af4b469d8765b1c5c18" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40035 } } peers { permanent_uuid: "e7228cda787344cda42bcd27a6c53eb3" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 33967 } }
I20260620 18:32:01.451745 11813 raft_consensus.cc:374] T 7674db37693b4e63976fcd1cb332d74a P 392cecf516d34c09b7c59f0e7f6fbf46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:01.451874 11813 raft_consensus.cc:729] T 7674db37693b4e63976fcd1cb332d74a P 392cecf516d34c09b7c59f0e7f6fbf46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 392cecf516d34c09b7c59f0e7f6fbf46, State: Initialized, Role: FOLLOWER
I20260620 18:32:01.452467 11813 consensus_queue.cc:260] T 7674db37693b4e63976fcd1cb332d74a P 392cecf516d34c09b7c59f0e7f6fbf46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392cecf516d34c09b7c59f0e7f6fbf46" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 44163 } } peers { permanent_uuid: "b468ca447ffb4af4b469d8765b1c5c18" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40035 } } peers { permanent_uuid: "e7228cda787344cda42bcd27a6c53eb3" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 33967 } }
I20260620 18:32:01.453548 11661 heartbeater.cc:503] Master 127.10.131.62:38863 was elected leader, sending a full tablet report...
I20260620 18:32:01.454180 11813 ts_tablet_manager.cc:1428] T 7674db37693b4e63976fcd1cb332d74a P 392cecf516d34c09b7c59f0e7f6fbf46: Time spent starting tablet: real 0.013s user 0.006s sys 0.006s
I20260620 18:32:01.532006 11820 raft_consensus.cc:482] T 7674db37693b4e63976fcd1cb332d74a P 392cecf516d34c09b7c59f0e7f6fbf46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:32:01.532353 11820 raft_consensus.cc:504] T 7674db37693b4e63976fcd1cb332d74a P 392cecf516d34c09b7c59f0e7f6fbf46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392cecf516d34c09b7c59f0e7f6fbf46" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 44163 } } peers { permanent_uuid: "b468ca447ffb4af4b469d8765b1c5c18" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40035 } } peers { permanent_uuid: "e7228cda787344cda42bcd27a6c53eb3" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 33967 } }
I20260620 18:32:01.534097 11820 leader_election.cc:288] T 7674db37693b4e63976fcd1cb332d74a P 392cecf516d34c09b7c59f0e7f6fbf46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b468ca447ffb4af4b469d8765b1c5c18 (127.10.131.3:40035), e7228cda787344cda42bcd27a6c53eb3 (127.10.131.1:33967)
I20260620 18:32:01.543546 11811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:01.534662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:34177 (local address 127.10.131.1:33967)
0620 18:32:01.534928 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:01.534937 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:01.535491 (+ 554us) server_negotiation.cc:407] Connection header received
0620 18:32:01.535720 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:01.535725 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:01.535802 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:01.535904 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:01.536709 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:01.537894 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:01.540994 (+ 3100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:01.542206 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:01.542369 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:01.542442 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:01.543101 (+ 659us) server_negotiation.cc:299] Negotiation successful
0620 18:32:01.543250 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:32:01.543546 11809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:01.534694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:60277 (local address 127.10.131.3:40035)
0620 18:32:01.534929 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:01.534936 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:01.535562 (+ 626us) server_negotiation.cc:407] Connection header received
0620 18:32:01.535719 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:01.535725 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:01.535802 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:01.535904 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:01.536709 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:01.537894 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:01.540992 (+ 3098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:01.542206 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:01.542369 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:01.542442 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:01.543101 (+ 659us) server_negotiation.cc:299] Negotiation successful
0620 18:32:01.543250 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 18:32:01.543731 11821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:01.534385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:40035 (local address 127.10.131.2:60277)
0620 18:32:01.535441 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:01.535463 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:01.535575 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:01.535971 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:01.535977 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:01.536078 (+ 101us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:01.536560 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:01.536568 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:01.538039 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:01.538043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:01.540790 (+ 2747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:01.540805 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:01.542512 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:01.542516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:01.542805 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:01.542922 (+ 117us) client_negotiation.cc:769] Sending connection context
0620 18:32:01.543022 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 18:32:01.543250 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":930,"mutex_wait_us":266,"spinlock_wait_cycles":41216,"thread_start_us":750,"threads_started":1}
I20260620 18:32:01.544091 11822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:01.534367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:33967 (local address 127.10.131.2:34177)
0620 18:32:01.535344 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:01.535368 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:01.535557 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:01.535971 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:01.535977 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:01.536018 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:01.536533 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:01.536554 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:01.538038 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:01.538043 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:01.540792 (+ 2749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:01.540805 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:01.542509 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:01.542516 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:01.542810 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:01.542922 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 18:32:01.543022 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 18:32:01.543250 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"mutex_wait_us":340,"thread_start_us":538,"threads_started":1}
I20260620 18:32:01.544952 11749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7674db37693b4e63976fcd1cb332d74a" candidate_uuid: "392cecf516d34c09b7c59f0e7f6fbf46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b468ca447ffb4af4b469d8765b1c5c18" is_pre_election: true
I20260620 18:32:01.545159 11482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7674db37693b4e63976fcd1cb332d74a" candidate_uuid: "392cecf516d34c09b7c59f0e7f6fbf46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7228cda787344cda42bcd27a6c53eb3" is_pre_election: true
I20260620 18:32:01.545611 11749 raft_consensus.cc:2454] T 7674db37693b4e63976fcd1cb332d74a P b468ca447ffb4af4b469d8765b1c5c18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 392cecf516d34c09b7c59f0e7f6fbf46 in term 0.
I20260620 18:32:01.545708 11482 raft_consensus.cc:2454] T 7674db37693b4e63976fcd1cb332d74a P e7228cda787344cda42bcd27a6c53eb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 392cecf516d34c09b7c59f0e7f6fbf46 in term 0.
I20260620 18:32:01.546451 11550 leader_election.cc:302] T 7674db37693b4e63976fcd1cb332d74a P 392cecf516d34c09b7c59f0e7f6fbf46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 392cecf516d34c09b7c59f0e7f6fbf46, b468ca447ffb4af4b469d8765b1c5c18, e7228cda787344cda42bcd27a6c53eb3; no voters:
I20260620 18:32:01.547053 11820 raft_consensus.cc:2781] T 7674db37693b4e63976fcd1cb332d74a P 392cecf516d34c09b7c59f0e7f6fbf46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:32:01.547224 11820 raft_consensus.cc:482] T 7674db37693b4e63976fcd1cb332d74a P 392cecf516d34c09b7c59f0e7f6fbf46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:32:01.547340 11820 raft_consensus.cc:3037] T 7674db37693b4e63976fcd1cb332d74a P 392cecf516d34c09b7c59f0e7f6fbf46 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:01.549868 11820 raft_consensus.cc:504] T 7674db37693b4e63976fcd1cb332d74a P 392cecf516d34c09b7c59f0e7f6fbf46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392cecf516d34c09b7c59f0e7f6fbf46" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 44163 } } peers { permanent_uuid: "b468ca447ffb4af4b469d8765b1c5c18" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40035 } } peers { permanent_uuid: "e7228cda787344cda42bcd27a6c53eb3" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 33967 } }
I20260620 18:32:01.550555 11820 leader_election.cc:288] T 7674db37693b4e63976fcd1cb332d74a P 392cecf516d34c09b7c59f0e7f6fbf46 [CANDIDATE]: Term 1 election: Requested vote from peers b468ca447ffb4af4b469d8765b1c5c18 (127.10.131.3:40035), e7228cda787344cda42bcd27a6c53eb3 (127.10.131.1:33967)
I20260620 18:32:01.550879 11749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7674db37693b4e63976fcd1cb332d74a" candidate_uuid: "392cecf516d34c09b7c59f0e7f6fbf46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b468ca447ffb4af4b469d8765b1c5c18"
I20260620 18:32:01.551108 11749 raft_consensus.cc:3037] T 7674db37693b4e63976fcd1cb332d74a P b468ca447ffb4af4b469d8765b1c5c18 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:01.551100 11482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7674db37693b4e63976fcd1cb332d74a" candidate_uuid: "392cecf516d34c09b7c59f0e7f6fbf46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7228cda787344cda42bcd27a6c53eb3"
I20260620 18:32:01.551297 11482 raft_consensus.cc:3037] T 7674db37693b4e63976fcd1cb332d74a P e7228cda787344cda42bcd27a6c53eb3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:01.553617 11749 raft_consensus.cc:2454] T 7674db37693b4e63976fcd1cb332d74a P b468ca447ffb4af4b469d8765b1c5c18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 392cecf516d34c09b7c59f0e7f6fbf46 in term 1.
I20260620 18:32:01.553637 11482 raft_consensus.cc:2454] T 7674db37693b4e63976fcd1cb332d74a P e7228cda787344cda42bcd27a6c53eb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 392cecf516d34c09b7c59f0e7f6fbf46 in term 1.
I20260620 18:32:01.554008 11552 leader_election.cc:302] T 7674db37693b4e63976fcd1cb332d74a P 392cecf516d34c09b7c59f0e7f6fbf46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 392cecf516d34c09b7c59f0e7f6fbf46, b468ca447ffb4af4b469d8765b1c5c18; no voters:
I20260620 18:32:01.554316 11820 raft_consensus.cc:2781] T 7674db37693b4e63976fcd1cb332d74a P 392cecf516d34c09b7c59f0e7f6fbf46 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:01.554766 11820 raft_consensus.cc:686] T 7674db37693b4e63976fcd1cb332d74a P 392cecf516d34c09b7c59f0e7f6fbf46 [term 1 LEADER]: Becoming Leader. State: Replica: 392cecf516d34c09b7c59f0e7f6fbf46, State: Running, Role: LEADER
I20260620 18:32:01.555318 11820 consensus_queue.cc:237] T 7674db37693b4e63976fcd1cb332d74a P 392cecf516d34c09b7c59f0e7f6fbf46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392cecf516d34c09b7c59f0e7f6fbf46" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 44163 } } peers { permanent_uuid: "b468ca447ffb4af4b469d8765b1c5c18" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40035 } } peers { permanent_uuid: "e7228cda787344cda42bcd27a6c53eb3" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 33967 } }
I20260620 18:32:01.560065 11337 catalog_manager.cc:5495] T 7674db37693b4e63976fcd1cb332d74a P 392cecf516d34c09b7c59f0e7f6fbf46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 392cecf516d34c09b7c59f0e7f6fbf46 (127.10.131.2). New cstate: current_term: 1 leader_uuid: "392cecf516d34c09b7c59f0e7f6fbf46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392cecf516d34c09b7c59f0e7f6fbf46" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 44163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b468ca447ffb4af4b469d8765b1c5c18" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7228cda787344cda42bcd27a6c53eb3" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 33967 } health_report { overall_health: UNKNOWN } } }
W20260620 18:32:01.561447 11662 tablet.cc:2369] T 7674db37693b4e63976fcd1cb332d74a P 392cecf516d34c09b7c59f0e7f6fbf46: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:32:01.566319 11795 tablet.cc:2369] T 7674db37693b4e63976fcd1cb332d74a P b468ca447ffb4af4b469d8765b1c5c18: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:32:01.581787 11528 tablet.cc:2369] T 7674db37693b4e63976fcd1cb332d74a P e7228cda787344cda42bcd27a6c53eb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 18:32:01 dist-test-slave-cbn3 krb5kdc[11293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.1@KRBTEST.COM
I20260620 18:32:01.602283 11811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:01.587537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58414 (local address 127.10.131.1:33967)
0620 18:32:01.587736 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:01.587740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:01.588356 (+ 616us) server_negotiation.cc:407] Connection header received
0620 18:32:01.588625 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:01.588630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:01.588716 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:01.588828 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:01.590307 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:01.591033 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:01.592351 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:01.592802 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:01.592979 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:01.596733 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:01.596746 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:01.596774 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:01.596804 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:01.599530 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:01.600159 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:01.600168 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:01.600171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:01.600300 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:01.600668 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:01.600674 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:01.600677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:01.601124 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:01.601415 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:01.601732 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 18:32:01.601957 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
Jun 20 18:32:01 dist-test-slave-cbn3 krb5kdc[11293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980321, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 18:32:01 dist-test-slave-cbn3 krb5kdc[11293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980321, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.131.1@KRBTEST.COM
I20260620 18:32:01.648674 11811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:01.635099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58428 (local address 127.10.131.1:33967)
0620 18:32:01.635364 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:01.635386 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:01.635576 (+ 190us) server_negotiation.cc:407] Connection header received
0620 18:32:01.635717 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:01.635721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:01.635783 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:01.635918 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:01.637229 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:01.637963 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:01.639239 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:01.639676 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:01.639798 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:01.643418 (+ 3620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:01.643423 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:01.643426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:01.643457 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:01.646202 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:01.646891 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:01.646895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:01.646897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:01.647023 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:01.647414 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:01.647417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:01.647419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:01.647717 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:01.647936 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:01.648166 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 18:32:01.648318 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
Jun 20 18:32:01 dist-test-slave-cbn3 krb5kdc[11293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 18:32:01 dist-test-slave-cbn3 krb5kdc[11293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.1@KRBTEST.COM
I20260620 18:32:01.693138 11811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:01.679277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58430 (local address 127.10.131.1:33967)
0620 18:32:01.679543 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:01.679550 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:01.679770 (+ 220us) server_negotiation.cc:407] Connection header received
0620 18:32:01.679988 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:01.679993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:01.680056 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:01.680141 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:01.681777 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:01.682491 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:01.683810 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:01.684233 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:01.684327 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:01.688237 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:01.688241 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:01.688245 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:01.688278 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:01.690780 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:01.691371 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:01.691375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:01.691377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:01.691487 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:01.691785 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:01.691787 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:01.691789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:01.692096 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:01.692264 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:01.692533 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 18:32:01.692701 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
W20260620 18:32:01.694814 11442 tablet_service.cc:2908] Rejecting scan request for tablet 7674db37693b4e63976fcd1cb332d74a: Uninitialized: clean time has not yet been initialized
I20260620 18:32:01.698791 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 11399
I20260620 18:32:01.710827 11811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:01.699627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58436 (local address 127.10.131.1:33967)
0620 18:32:01.699878 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:01.699882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:01.700224 (+ 342us) server_negotiation.cc:407] Connection header received
0620 18:32:01.700349 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:01.700353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:01.700430 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:01.700507 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:01.701704 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:01.702542 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:01.703695 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:01.704089 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:01.704189 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:01.706267 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:01.706272 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:01.706275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:01.706301 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:01.708743 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:01.709350 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:01.709353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:01.709355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:01.709462 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:01.709781 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:01.709783 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:01.709786 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:01.710031 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:01.710225 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:01.710417 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 18:32:01.710597 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260620 18:32:01.711447 11522 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:01.800374 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 11399
I20260620 18:32:01.821043 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 11533
Jun 20 18:32:01 dist-test-slave-cbn3 krb5kdc[11293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.2@KRBTEST.COM
I20260620 18:32:01.835503 11807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:01.821952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40718 (local address 127.10.131.2:44163)
0620 18:32:01.822172 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:01.822179 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:01.822442 (+ 263us) server_negotiation.cc:407] Connection header received
0620 18:32:01.822661 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:01.822665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:01.822729 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:01.822834 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:01.823957 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:01.824694 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:01.825955 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:01.826388 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:01.826541 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:01.830454 (+ 3913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:01.830464 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:01.830495 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:01.830525 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:01.833129 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:01.833755 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:01.833761 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:01.833764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:01.833906 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:01.834210 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:01.834213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:01.834215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:01.834535 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:01.834746 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:01.834955 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 18:32:01.835134 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:32:01.836210 11656 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:01.925858 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 11533
I20260620 18:32:01.942314 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 11666
Jun 20 18:32:01 dist-test-slave-cbn3 krb5kdc[11293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.3@KRBTEST.COM
I20260620 18:32:01.956889 11809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:01.943084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37314 (local address 127.10.131.3:40035)
0620 18:32:01.943269 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:01.943275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:01.943297 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:32:01.943371 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:01.943376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:01.943463 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:01.943573 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:01.944757 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:01.945546 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:01.946826 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:01.947205 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:01.947350 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:01.951442 (+ 4092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:01.951456 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:01.951472 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:01.951506 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:01.954407 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:01.955019 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:01.955026 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:01.955029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:01.955170 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:01.955561 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:01.955564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:01.955567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:01.955899 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:01.956112 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:01.956401 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 18:32:01.956588 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:32:01.957526 11789 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:02.043994 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 11666
I20260620 18:32:02.059845 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 11306
I20260620 18:32:02.061128 11368 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:02.167411 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 11306
2026-06-20T18:32:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4143 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:32:02 dist-test-slave-cbn3 krb5kdc[11842](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:32:02 dist-test-slave-cbn3 krb5kdc[11842](info): set up 2 sockets
Jun 20 18:32:02 dist-test-slave-cbn3 krb5kdc[11842](info): commencing operation
krb5kdc: starting...
W20260620 18:32:03.250394 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:32:03 dist-test-slave-cbn3 krb5kdc[11842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:32:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:32:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.131.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
I20260620 18:32:03.432435 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:45203
--webserver_interface=127.10.131.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:45933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:45203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:03.758201 11855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:03.758494 11855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:03.758582 11855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:03.765676 11855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:32:03.765774 11855 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 18:32:03.765815 11855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:03.765874 11855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:03.765924 11855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:32:03.765978 11855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:32:03.775998 11855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:45933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:45203
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:45203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:03.777235 11855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:03.778447 11855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:03.789706 11860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:03.789762 11863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:03.789790 11861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:03.790894 11855 server_base.cc:1034] running on GCE node
I20260620 18:32:03.793784 11855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:03.795169 11855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:03.796494 11855 hybrid_clock.cc:648] HybridClock initialized: now 1781980323796433 us; error 29 us; skew 500 ppm
I20260620 18:32:03.796877 11855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:03 dist-test-slave-cbn3 krb5kdc[11842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980323, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:03.800026 11855 init.cc:377] Logged in from keytab as kudu/127.10.131.62@KRBTEST.COM (short username kudu)
I20260620 18:32:03.801466 11855 webserver.cc:460] Webserver started at http://127.10.131.62:44147/ using document root <none> and password file <none>
I20260620 18:32:03.802085 11855 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:03.802174 11855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:03.802431 11855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:03.805291 11855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "c1cb409d78ff4d0d8c15e24ede3ff4bd"
format_stamp: "Formatted at 2026-06-20 18:32:03 on dist-test-slave-cbn3"
server_key: "032d3fa52346751fc6c03a4b8515afec"
server_key_iv: "cbc83250dc5d78abd74fbff845679578"
server_key_version: "encryptionkey@0"
I20260620 18:32:03.806026 11855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "c1cb409d78ff4d0d8c15e24ede3ff4bd"
format_stamp: "Formatted at 2026-06-20 18:32:03 on dist-test-slave-cbn3"
server_key: "032d3fa52346751fc6c03a4b8515afec"
server_key_iv: "cbc83250dc5d78abd74fbff845679578"
server_key_version: "encryptionkey@0"
I20260620 18:32:03.810922 11855 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:32:03.814150 11870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:03.815460 11855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:32:03.815639 11855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "c1cb409d78ff4d0d8c15e24ede3ff4bd"
format_stamp: "Formatted at 2026-06-20 18:32:03 on dist-test-slave-cbn3"
server_key: "032d3fa52346751fc6c03a4b8515afec"
server_key_iv: "cbc83250dc5d78abd74fbff845679578"
server_key_version: "encryptionkey@0"
I20260620 18:32:03.815821 11855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:03.853325 11855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:03.857633 11855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:03.857923 11855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:03.879534 11855 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:45203
I20260620 18:32:03.879617 11931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:45203 every 8 connection(s)
I20260620 18:32:03.880898 11855 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:32:03.884503 11932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:03.889345 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 11855
I20260620 18:32:03.889587 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:32:03.890249 10764 external_mini_cluster.cc:1428] Setting key 2907158f096c5f35ecea1061af3f85c6
I20260620 18:32:03.895275 11932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1cb409d78ff4d0d8c15e24ede3ff4bd: Bootstrap starting.
I20260620 18:32:03.898901 11932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1cb409d78ff4d0d8c15e24ede3ff4bd: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:03.900081 11932 log.cc:824] T 00000000000000000000000000000000 P c1cb409d78ff4d0d8c15e24ede3ff4bd: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:03.903113 11932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1cb409d78ff4d0d8c15e24ede3ff4bd: No bootstrap required, opened a new log
Jun 20 18:32:03 dist-test-slave-cbn3 krb5kdc[11842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:32:03.913486 11932 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1cb409d78ff4d0d8c15e24ede3ff4bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1cb409d78ff4d0d8c15e24ede3ff4bd" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45203 } }
I20260620 18:32:03.913882 11935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:03.892361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55936 (local address 127.10.131.62:45203)
0620 18:32:03.893235 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:03.893247 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:03.893289 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:32:03.894927 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:03.894952 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:03.895312 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:03.895677 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:03.897413 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:03.898704 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:03.900451 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:03.901254 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:03.901444 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:03.905581 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:03.905592 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:03.905610 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:03.905669 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:03.909621 (+ 3952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:03.910226 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:03.910234 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:03.910243 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:03.910451 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:03.910772 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:03.910783 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:03.910787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:03.911468 (+ 681us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:03.911771 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:03.912162 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 18:32:03.912590 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":419,"threads_started":1}
I20260620 18:32:03.913908 11932 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1cb409d78ff4d0d8c15e24ede3ff4bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:03.914085 11932 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1cb409d78ff4d0d8c15e24ede3ff4bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1cb409d78ff4d0d8c15e24ede3ff4bd, State: Initialized, Role: FOLLOWER
I20260620 18:32:03.914742 11932 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1cb409d78ff4d0d8c15e24ede3ff4bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1cb409d78ff4d0d8c15e24ede3ff4bd" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45203 } }
I20260620 18:32:03.915019 11932 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1cb409d78ff4d0d8c15e24ede3ff4bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:32:03.915155 11932 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1cb409d78ff4d0d8c15e24ede3ff4bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:32:03.915356 11932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1cb409d78ff4d0d8c15e24ede3ff4bd [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:03.917795 11932 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1cb409d78ff4d0d8c15e24ede3ff4bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1cb409d78ff4d0d8c15e24ede3ff4bd" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45203 } }
I20260620 18:32:03.918272 11932 leader_election.cc:302] T 00000000000000000000000000000000 P c1cb409d78ff4d0d8c15e24ede3ff4bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1cb409d78ff4d0d8c15e24ede3ff4bd; no voters:
I20260620 18:32:03.918936 11932 leader_election.cc:288] T 00000000000000000000000000000000 P c1cb409d78ff4d0d8c15e24ede3ff4bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:32:03.919030 11937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1cb409d78ff4d0d8c15e24ede3ff4bd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:03.919883 11937 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1cb409d78ff4d0d8c15e24ede3ff4bd [term 1 LEADER]: Becoming Leader. State: Replica: c1cb409d78ff4d0d8c15e24ede3ff4bd, State: Running, Role: LEADER
I20260620 18:32:03.920393 11937 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1cb409d78ff4d0d8c15e24ede3ff4bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1cb409d78ff4d0d8c15e24ede3ff4bd" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45203 } }
I20260620 18:32:03.921100 11932 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1cb409d78ff4d0d8c15e24ede3ff4bd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:32:03.923089 11939 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1cb409d78ff4d0d8c15e24ede3ff4bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1cb409d78ff4d0d8c15e24ede3ff4bd. Latest consensus state: current_term: 1 leader_uuid: "c1cb409d78ff4d0d8c15e24ede3ff4bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1cb409d78ff4d0d8c15e24ede3ff4bd" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45203 } } }
I20260620 18:32:03.923533 11939 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1cb409d78ff4d0d8c15e24ede3ff4bd [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:03.923996 11938 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1cb409d78ff4d0d8c15e24ede3ff4bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1cb409d78ff4d0d8c15e24ede3ff4bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1cb409d78ff4d0d8c15e24ede3ff4bd" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45203 } } }
I20260620 18:32:03.924338 11938 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1cb409d78ff4d0d8c15e24ede3ff4bd [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:03.925803 11943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:32:03.931499 11943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:32:03.939954 11943 catalog_manager.cc:1347] Generated new cluster ID: 50a721a3ef7b49b3a9e8103b0c886c0a
I20260620 18:32:03.940078 11943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:32:03.970278 11943 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:32:03.971575 11943 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:32:03.980827 11943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1cb409d78ff4d0d8c15e24ede3ff4bd: Generated new TSK 0
I20260620 18:32:03.981738 11943 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.131.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab.
Entry for principal kudu/127.10.131.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab.
I20260620 18:32:04.032243 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.1:0
--local_ip_for_outbound_sockets=127.10.131.1
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:45203
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:45933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:04.368330 11958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:04.368595 11958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:04.368759 11958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:04.375936 11958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:04.376032 11958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:04.376111 11958 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:32:04.376190 11958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.1
I20260620 18:32:04.385761 11958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:45933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.131.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.131.62:45203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.1
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:04.386900 11958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:04.388230 11958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:04.401101 11963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:04.401199 11964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:04.401132 11966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:04.402251 11958 server_base.cc:1034] running on GCE node
I20260620 18:32:04.402842 11958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:04.403647 11958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:04.404928 11958 hybrid_clock.cc:648] HybridClock initialized: now 1781980324404868 us; error 50 us; skew 500 ppm
I20260620 18:32:04.405215 11958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:04 dist-test-slave-cbn3 krb5kdc[11842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980324, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:04.408439 11958 init.cc:377] Logged in from keytab as kudu/127.10.131.1@KRBTEST.COM (short username kudu)
I20260620 18:32:04.410002 11958 webserver.cc:460] Webserver started at http://127.10.131.1:33261/ using document root <none> and password file <none>
I20260620 18:32:04.410666 11958 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:04.410797 11958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:04.411100 11958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:04.413897 11958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-0/data/instance:
uuid: "8d97b360ab544c8ba86d44fc332d7312"
format_stamp: "Formatted at 2026-06-20 18:32:04 on dist-test-slave-cbn3"
server_key: "147d9f49ef4fe18675bc65dd819c398a"
server_key_iv: "30191b555283618d0d7e6ac35f5f47b1"
server_key_version: "encryptionkey@0"
I20260620 18:32:04.414650 11958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance:
uuid: "8d97b360ab544c8ba86d44fc332d7312"
format_stamp: "Formatted at 2026-06-20 18:32:04 on dist-test-slave-cbn3"
server_key: "147d9f49ef4fe18675bc65dd819c398a"
server_key_iv: "30191b555283618d0d7e6ac35f5f47b1"
server_key_version: "encryptionkey@0"
I20260620 18:32:04.419585 11958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:32:04.423202 11973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:04.424516 11958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:32:04.424724 11958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-0/wal
uuid: "8d97b360ab544c8ba86d44fc332d7312"
format_stamp: "Formatted at 2026-06-20 18:32:04 on dist-test-slave-cbn3"
server_key: "147d9f49ef4fe18675bc65dd819c398a"
server_key_iv: "30191b555283618d0d7e6ac35f5f47b1"
server_key_version: "encryptionkey@0"
I20260620 18:32:04.425020 11958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:04.439304 11958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:04.443324 11958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:04.443604 11958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:04.446301 11958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:04.446406 11958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:04.446616 11958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:04.446688 11958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:32:04 dist-test-slave-cbn3 krb5kdc[11842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980324, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:32:04.469359 11981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:04.446714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:46325 (local address 127.10.131.62:45203)
0620 18:32:04.447187 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:04.447196 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:04.448408 (+ 1212us) server_negotiation.cc:407] Connection header received
0620 18:32:04.450547 (+ 2139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:04.450555 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:04.450664 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:04.450792 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:04.452938 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:04.453988 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:04.457205 (+ 3217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:04.457827 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:04.457976 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:04.462899 (+ 4923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:04.462908 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:04.462915 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:04.462966 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:04.466366 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:04.467063 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:04.467069 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:04.467072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:04.467199 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:04.467635 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:04.467640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:04.467643 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:04.467919 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:04.468072 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:04.468748 (+ 676us) server_negotiation.cc:299] Negotiation successful
0620 18:32:04.469002 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":247,"threads_started":1}
I20260620 18:32:04.470839 11983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:04.447395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:45203 (local address 127.10.131.1:46325)
0620 18:32:04.448198 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:04.448245 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:04.450154 (+ 1909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:04.451066 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:04.451083 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:04.451892 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:04.452731 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:04.452750 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:04.454183 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:04.454191 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:04.457026 (+ 2835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:04.457044 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:04.458140 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:04.458151 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:04.458521 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:04.459453 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:04.459478 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:04.462645 (+ 3167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:04.466509 (+ 3864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:04.466517 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:04.466533 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:04.466933 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:04.467315 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:04.467321 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:04.467325 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:04.467515 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:04.468052 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:04.468059 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:04.468386 (+ 327us) client_negotiation.cc:769] Sending connection context
0620 18:32:04.468662 (+ 276us) client_negotiation.cc:240] Negotiation successful
0620 18:32:04.469222 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":367,"threads_started":1}
I20260620 18:32:04.496244 12052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:04.477410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:45203 (local address 127.10.131.1:37005)
0620 18:32:04.481083 (+ 3673us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:04.481106 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:04.481261 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:04.481647 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:04.481653 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:04.482397 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:04.482869 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:04.482886 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:04.484160 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:04.484168 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:04.485915 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:04.485926 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:04.487168 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:04.487177 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:04.487325 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:04.488013 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:04.488029 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:04.489087 (+ 1058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:04.493216 (+ 4129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:04.493225 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:04.493232 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:04.493944 (+ 712us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:04.494383 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:04.494387 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:04.494390 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:04.494534 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:04.495023 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:04.495028 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:04.495249 (+ 221us) client_negotiation.cc:769] Sending connection context
0620 18:32:04.495456 (+ 207us) client_negotiation.cc:240] Negotiation successful
0620 18:32:04.495772 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3517,"thread_start_us":3324,"threads_started":1}
I20260620 18:32:04.496356 11958 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.1:39489
I20260620 18:32:04.496393 12093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.1:39489 every 8 connection(s)
I20260620 18:32:04.496832 11981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:04.477237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:37005 (local address 127.10.131.62:45203)
0620 18:32:04.477414 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:04.477420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:04.481191 (+ 3771us) server_negotiation.cc:407] Connection header received
0620 18:32:04.481406 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:04.481411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:04.481499 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:04.481601 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:04.483031 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:04.483949 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:04.486114 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:04.486862 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:04.487010 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:04.489310 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:04.489314 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:04.489317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:04.489347 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:04.493034 (+ 3687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:04.494083 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:04.494091 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:04.494096 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:04.494260 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:04.494642 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:04.494646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:04.494648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:04.494905 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:04.495059 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:04.496266 (+ 1207us) server_negotiation.cc:299] Negotiation successful
0620 18:32:04.496515 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:32:04.497694 11958 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
I20260620 18:32:04.499825 12094 heartbeater.cc:347] Connected to a master server at 127.10.131.62:45203
I20260620 18:32:04.500061 12094 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:04.500689 12094 heartbeater.cc:511] Master 127.10.131.62:45203 requested a full tablet report, sending...
I20260620 18:32:04.501657 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 11958
I20260620 18:32:04.501880 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance
I20260620 18:32:04.502557 10764 external_mini_cluster.cc:1428] Setting key 3e57b563c565cbac5f964ff7abb613a0
I20260620 18:32:04.502796 11887 ts_manager.cc:194] Registered new tserver with Master: 8d97b360ab544c8ba86d44fc332d7312 (127.10.131.1:39489)
I20260620 18:32:04.504597 11887 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.1@KRBTEST.COM'} at 127.10.131.1:46325
WARNING: no policy specified for kudu/127.10.131.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab.
Entry for principal kudu/127.10.131.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab.
I20260620 18:32:04.540324 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.2:0
--local_ip_for_outbound_sockets=127.10.131.2
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:45203
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:45933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:04.857438 12098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:04.857767 12098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:04.857942 12098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:04.865197 12098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:04.865340 12098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:04.865486 12098 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:32:04.865620 12098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.2
I20260620 18:32:04.875458 12098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:45933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.131.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.131.62:45203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.2
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:04.876767 12098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:04.878141 12098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:04.891508 12104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:04.891466 12106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:04.892284 12103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:04.892570 12098 server_base.cc:1034] running on GCE node
I20260620 18:32:04.893280 12098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:04.894311 12098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:04.895543 12098 hybrid_clock.cc:648] HybridClock initialized: now 1781980324895482 us; error 30 us; skew 500 ppm
I20260620 18:32:04.895867 12098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:04 dist-test-slave-cbn3 krb5kdc[11842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980324, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:04.902944 12098 init.cc:377] Logged in from keytab as kudu/127.10.131.2@KRBTEST.COM (short username kudu)
I20260620 18:32:04.904410 12098 webserver.cc:460] Webserver started at http://127.10.131.2:36191/ using document root <none> and password file <none>
I20260620 18:32:04.905185 12098 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:04.905272 12098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:04.905646 12098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:04.908600 12098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-1/data/instance:
uuid: "ab3b9330a0544503a5fd00fa7f77db11"
format_stamp: "Formatted at 2026-06-20 18:32:04 on dist-test-slave-cbn3"
server_key: "dd7be6d4dc0b60a13b10b2210c6a9663"
server_key_iv: "6ca15a55321fe72de764670f174f8b56"
server_key_version: "encryptionkey@0"
I20260620 18:32:04.909457 12098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance:
uuid: "ab3b9330a0544503a5fd00fa7f77db11"
format_stamp: "Formatted at 2026-06-20 18:32:04 on dist-test-slave-cbn3"
server_key: "dd7be6d4dc0b60a13b10b2210c6a9663"
server_key_iv: "6ca15a55321fe72de764670f174f8b56"
server_key_version: "encryptionkey@0"
I20260620 18:32:04.914376 12098 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 18:32:04.917627 12113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:04.919008 12098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:32:04.919190 12098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-1/wal
uuid: "ab3b9330a0544503a5fd00fa7f77db11"
format_stamp: "Formatted at 2026-06-20 18:32:04 on dist-test-slave-cbn3"
server_key: "dd7be6d4dc0b60a13b10b2210c6a9663"
server_key_iv: "6ca15a55321fe72de764670f174f8b56"
server_key_version: "encryptionkey@0"
I20260620 18:32:04.919411 12098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:04.942560 12098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:04.946867 12098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:04.947144 12098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:04.949847 12098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:04.949955 12098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:04.950151 12098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:04.950242 12098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:32:04 dist-test-slave-cbn3 krb5kdc[11842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980324, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:32:04.973995 11981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:04.950375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:59641 (local address 127.10.131.62:45203)
0620 18:32:04.950631 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:04.950639 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:04.952073 (+ 1434us) server_negotiation.cc:407] Connection header received
0620 18:32:04.954232 (+ 2159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:04.954239 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:04.954339 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:04.954479 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:04.956896 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:04.957983 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:04.960543 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:04.961127 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:04.961254 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:04.966485 (+ 5231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:04.966494 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:04.966500 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:04.966546 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:04.970029 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:04.971018 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:04.971027 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:04.971033 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:04.971230 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:04.971736 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:04.971744 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:04.971750 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:04.972259 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:04.972448 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:04.973405 (+ 957us) server_negotiation.cc:299] Negotiation successful
0620 18:32:04.973630 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:32:04.975037 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:04.951033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:45203 (local address 127.10.131.2:59641)
0620 18:32:04.951881 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:04.951929 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:04.953851 (+ 1922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:04.954826 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:04.954839 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:04.955619 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:04.956674 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:04.956704 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:04.958165 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:04.958171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:04.960390 (+ 2219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:04.960402 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:04.961395 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:04.961407 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:04.961618 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:04.962543 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:04.962583 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:04.966176 (+ 3593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:04.970177 (+ 4001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:04.970190 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:04.970211 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:04.970874 (+ 663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:04.971394 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:04.971398 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:04.971401 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:04.971608 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:04.972455 (+ 847us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:04.972461 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:04.972952 (+ 491us) client_negotiation.cc:769] Sending connection context
0620 18:32:04.973292 (+ 340us) client_negotiation.cc:240] Negotiation successful
0620 18:32:04.973815 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":365,"threads_started":1}
I20260620 18:32:04.997324 11981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:04.980919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:54961 (local address 127.10.131.62:45203)
0620 18:32:04.981099 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:04.981105 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:04.981825 (+ 720us) server_negotiation.cc:407] Connection header received
0620 18:32:04.982034 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:04.982042 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:04.982143 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:04.982242 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:04.983259 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:04.984447 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:04.987130 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:04.987746 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:04.987909 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:04.990542 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:04.990550 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:04.990558 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:04.990604 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:04.993503 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:04.994684 (+ 1181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:04.994692 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:04.994698 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:04.994903 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:04.995411 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:04.995416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:04.995420 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:04.995833 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:04.995995 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:04.996632 (+ 637us) server_negotiation.cc:299] Negotiation successful
0620 18:32:04.996931 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:32:04.997972 12189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:04.981003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:45203 (local address 127.10.131.2:54961)
0620 18:32:04.981705 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:04.981721 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:04.981893 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:04.982411 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:04.982414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:04.982839 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:04.983113 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:04.983121 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:04.984624 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:04.984632 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:04.986964 (+ 2332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:04.986988 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:04.988004 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:04.988009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:04.988272 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:04.988973 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:04.989012 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:04.990336 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:04.993635 (+ 3299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:04.993644 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:04.993656 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:04.994524 (+ 868us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:04.995048 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:04.995056 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:04.995062 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:04.995250 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:04.995972 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:04.995980 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:04.996251 (+ 271us) client_negotiation.cc:769] Sending connection context
0620 18:32:04.996470 (+ 219us) client_negotiation.cc:240] Negotiation successful
0620 18:32:04.996921 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":411,"threads_started":1}
I20260620 18:32:04.999912 12232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.2:40403 every 8 connection(s)
I20260620 18:32:04.999946 12098 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.2:40403
I20260620 18:32:05.001507 12098 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
I20260620 18:32:05.003595 12233 heartbeater.cc:347] Connected to a master server at 127.10.131.62:45203
I20260620 18:32:05.003813 12233 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:05.004426 12233 heartbeater.cc:511] Master 127.10.131.62:45203 requested a full tablet report, sending...
I20260620 18:32:05.005803 11887 ts_manager.cc:194] Registered new tserver with Master: ab3b9330a0544503a5fd00fa7f77db11 (127.10.131.2:40403)
I20260620 18:32:05.006609 11887 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.2@KRBTEST.COM'} at 127.10.131.2:59641
I20260620 18:32:05.008126 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 12098
I20260620 18:32:05.008351 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance
I20260620 18:32:05.008803 10764 external_mini_cluster.cc:1428] Setting key f751ccfef6214a8b113a980b2640bc49
WARNING: no policy specified for kudu/127.10.131.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab.
Entry for principal kudu/127.10.131.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab.
I20260620 18:32:05.045966 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.3:0
--local_ip_for_outbound_sockets=127.10.131.3
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:45203
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:45933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:05.379739 12237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:05.380055 12237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:05.380220 12237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:05.387756 12237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:05.387892 12237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:05.388047 12237 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:32:05.388171 12237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.3
I20260620 18:32:05.398375 12237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:45933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.131.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.131.62:45203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.3
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:05.399677 12237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:05.401188 12237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:05.414850 12245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:05.414896 12243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:05.414935 12242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:05.415659 12237 server_base.cc:1034] running on GCE node
I20260620 18:32:05.416316 12237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:05.417315 12237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:05.418556 12237 hybrid_clock.cc:648] HybridClock initialized: now 1781980325418501 us; error 34 us; skew 500 ppm
I20260620 18:32:05.418880 12237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:05 dist-test-slave-cbn3 krb5kdc[11842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980325, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:05.422216 12237 init.cc:377] Logged in from keytab as kudu/127.10.131.3@KRBTEST.COM (short username kudu)
I20260620 18:32:05.423676 12237 webserver.cc:460] Webserver started at http://127.10.131.3:33361/ using document root <none> and password file <none>
I20260620 18:32:05.424412 12237 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:05.424544 12237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:05.424930 12237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:05.428081 12237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-2/data/instance:
uuid: "57e470bede444347bb8661831e02d6b2"
format_stamp: "Formatted at 2026-06-20 18:32:05 on dist-test-slave-cbn3"
server_key: "ccab6819b5faf9a9c1ceae97fb17d5de"
server_key_iv: "60faf0d560b3d2a2a41292fc2ccbd566"
server_key_version: "encryptionkey@0"
I20260620 18:32:05.428968 12237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance:
uuid: "57e470bede444347bb8661831e02d6b2"
format_stamp: "Formatted at 2026-06-20 18:32:05 on dist-test-slave-cbn3"
server_key: "ccab6819b5faf9a9c1ceae97fb17d5de"
server_key_iv: "60faf0d560b3d2a2a41292fc2ccbd566"
server_key_version: "encryptionkey@0"
I20260620 18:32:05.433885 12237 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:32:05.437309 12252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:05.438655 12237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 18:32:05.438841 12237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-2/wal
uuid: "57e470bede444347bb8661831e02d6b2"
format_stamp: "Formatted at 2026-06-20 18:32:05 on dist-test-slave-cbn3"
server_key: "ccab6819b5faf9a9c1ceae97fb17d5de"
server_key_iv: "60faf0d560b3d2a2a41292fc2ccbd566"
server_key_version: "encryptionkey@0"
I20260620 18:32:05.439075 12237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:05.476734 12237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:05.481374 12237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:05.481668 12237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:05.484315 12237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:05.484457 12237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:05.484634 12237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:05.484717 12237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:32:05 dist-test-slave-cbn3 krb5kdc[11842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980325, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:32:05.507833 12094 heartbeater.cc:503] Master 127.10.131.62:45203 was elected leader, sending a full tablet report...
I20260620 18:32:05.512629 11981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:05.485242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:38299 (local address 127.10.131.62:45203)
0620 18:32:05.485440 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:05.485446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:05.487030 (+ 1584us) server_negotiation.cc:407] Connection header received
0620 18:32:05.490225 (+ 3195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:05.490233 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:05.490333 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:05.490466 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:05.493245 (+ 2779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:05.494214 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:05.497543 (+ 3329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:05.498150 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:05.498283 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:05.503675 (+ 5392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:05.503685 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:05.503693 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:05.503754 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:05.508032 (+ 4278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:05.509031 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:05.509036 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:05.509039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:05.509201 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:05.509717 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:05.509726 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:05.509732 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:05.510160 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:05.510327 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:05.511879 (+ 1552us) server_negotiation.cc:299] Negotiation successful
0620 18:32:05.512188 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 18:32:05.514690 12261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:05.485854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:45203 (local address 127.10.131.3:38299)
0620 18:32:05.486822 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:05.486889 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:05.489781 (+ 2892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:05.490774 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:05.490789 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:05.491849 (+ 1060us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:05.493019 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:05.493055 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:05.494451 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:05.494459 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:05.497337 (+ 2878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:05.497359 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:05.498775 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:05.498785 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:05.499094 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:05.499990 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:05.500034 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:05.503371 (+ 3337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:05.508198 (+ 4827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:05.508207 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:05.508224 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:05.508813 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:05.509343 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:05.509347 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:05.509351 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:05.509595 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:05.510636 (+ 1041us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:05.510647 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:05.511295 (+ 648us) client_negotiation.cc:769] Sending connection context
0620 18:32:05.511709 (+ 414us) client_negotiation.cc:240] Negotiation successful
0620 18:32:05.512470 (+ 761us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":414,"threads_started":1}
I20260620 18:32:05.536702 11981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:05.521059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:38639 (local address 127.10.131.62:45203)
0620 18:32:05.521524 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:05.521532 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:05.521856 (+ 324us) server_negotiation.cc:407] Connection header received
0620 18:32:05.522026 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:05.522031 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:05.522109 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:05.522214 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:05.523221 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:05.524044 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:05.526067 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:05.526910 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:05.527077 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:05.529604 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:05.529609 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:05.529612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:05.529642 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:05.533054 (+ 3412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:05.534200 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:05.534209 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:05.534212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:05.534446 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:05.534966 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:05.534973 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:05.534976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:05.535287 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:05.535461 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:05.536095 (+ 634us) server_negotiation.cc:299] Negotiation successful
0620 18:32:05.536316 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294}
I20260620 18:32:05.536865 12334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:05.521145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:45203 (local address 127.10.131.3:38639)
0620 18:32:05.521727 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:05.521752 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:05.521898 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:05.522239 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:05.522243 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:05.522701 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:05.523081 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:05.523091 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:05.524201 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:05.524206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:05.525816 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:05.525832 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:05.527135 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:05.527141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:05.527259 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:05.527986 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:05.528019 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:05.529432 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:05.533238 (+ 3806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:05.533247 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:05.533254 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:05.534028 (+ 774us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:05.534611 (+ 583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:05.534616 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:05.534620 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:05.534804 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:05.535532 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:05.535538 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:05.535794 (+ 256us) client_negotiation.cc:769] Sending connection context
0620 18:32:05.535977 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 18:32:05.536333 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":393,"threads_started":1}
I20260620 18:32:05.538757 12237 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.3:37191
I20260620 18:32:05.538790 12371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.3:37191 every 8 connection(s)
I20260620 18:32:05.540278 12237 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
I20260620 18:32:05.542305 12372 heartbeater.cc:347] Connected to a master server at 127.10.131.62:45203
I20260620 18:32:05.542541 12372 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:05.543228 12372 heartbeater.cc:511] Master 127.10.131.62:45203 requested a full tablet report, sending...
I20260620 18:32:05.543560 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 12237
I20260620 18:32:05.543757 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance
I20260620 18:32:05.544181 10764 external_mini_cluster.cc:1428] Setting key e68142339fd0d383ebe484bdd13dfff4
I20260620 18:32:05.545480 11886 ts_manager.cc:194] Registered new tserver with Master: 57e470bede444347bb8661831e02d6b2 (127.10.131.3:37191)
I20260620 18:32:05.546686 11886 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.3@KRBTEST.COM'} at 127.10.131.3:38299
I20260620 18:32:05.547350 10764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 18:32:05 dist-test-slave-cbn3 krb5kdc[11842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980325, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 18:32:05 dist-test-slave-cbn3 krb5kdc[11842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980325, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:32:05.592723 11981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:05.578792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55944 (local address 127.10.131.62:45203)
0620 18:32:05.579020 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:05.579026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:05.579243 (+ 217us) server_negotiation.cc:407] Connection header received
0620 18:32:05.579466 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:05.579472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:05.579542 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:05.579648 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:05.581065 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:05.581765 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:05.583089 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:05.583524 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:05.583667 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:05.587404 (+ 3737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:05.587416 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:05.587419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:05.587484 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:05.590033 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:05.590710 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:05.590716 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:05.590720 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:05.590848 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:05.591266 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:05.591270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:05.591274 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:05.591536 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:05.591783 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:05.592094 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 18:32:05.592325 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 18:32:05.596705 11886 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:32:05.603458 11886 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:32:05.629447 12388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:05.616631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:40403 (local address 127.0.0.1:38174)
0620 18:32:05.619012 (+ 2381us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:05.619061 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:05.619269 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:05.620632 (+ 1363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:05.620644 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:05.620688 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:05.621254 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:05.621270 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:05.623245 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:05.623252 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:05.625671 (+ 2419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:05.625690 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:05.628299 (+ 2609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:05.628307 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:05.628525 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:05.628575 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 18:32:05.628720 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 18:32:05.628939 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2048,"thread_start_us":4377,"threads_started":1}
I20260620 18:32:05.629659 12386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:05.616833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38174 (local address 127.10.131.2:40403)
0620 18:32:05.620006 (+ 3173us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:05.620016 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:05.620045 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:32:05.620133 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:05.620138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:05.620406 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:05.620559 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:05.621552 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:05.623101 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:05.625839 (+ 2738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:05.628025 (+ 2186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:05.628199 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:05.628484 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:05.628815 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 18:32:05.629137 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3001,"thread_start_us":2887,"threads_started":1}
I20260620 18:32:05.630366 12387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:05.617533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52798 (local address 127.10.131.1:39489)
0620 18:32:05.618285 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:05.618298 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:05.621347 (+ 3049us) server_negotiation.cc:407] Connection header received
0620 18:32:05.621557 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:05.621564 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:05.621779 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:05.622277 (+ 498us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:05.623167 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:05.624722 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:05.627132 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:05.628289 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:05.628465 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:05.628666 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:05.629580 (+ 914us) server_negotiation.cc:299] Negotiation successful
0620 18:32:05.629747 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":495,"threads_started":1}
I20260620 18:32:05.631024 12385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:05.616954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:39489 (local address 127.0.0.1:52798)
0620 18:32:05.621233 (+ 4279us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:05.621247 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:05.621389 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:05.622493 (+ 1104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:05.622498 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:05.622522 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:05.622939 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:05.622949 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:05.624915 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:05.624919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:05.626985 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:05.626997 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:05.629038 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:05.629044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:05.629232 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:05.629260 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 18:32:05.629347 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 18:32:05.630659 (+ 1312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4177,"thread_start_us":1561,"threads_started":1}
I20260620 18:32:05.635353 12024 tablet_service.cc:1467] Processing CreateTablet for tablet 1695dab04b194158b55ed646e33d0ee2 (DEFAULT_TABLE table=test-table [id=3cd4b4a2c3c541d889ac5bb0afae48a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:05.636361 12163 tablet_service.cc:1467] Processing CreateTablet for tablet 1695dab04b194158b55ed646e33d0ee2 (DEFAULT_TABLE table=test-table [id=3cd4b4a2c3c541d889ac5bb0afae48a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:05.636932 12024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1695dab04b194158b55ed646e33d0ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:05.636946 12388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:05.622379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:37191 (local address 127.0.0.1:58778)
0620 18:32:05.629702 (+ 7323us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:05.629720 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:05.629821 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:05.630844 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:05.630849 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:05.630873 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:05.631048 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:05.631058 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:05.632591 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:05.632600 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:05.634167 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:05.634183 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:05.636191 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:05.636197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:05.636370 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:05.636408 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 18:32:05.636524 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 18:32:05.636657 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7200,"thread_start_us":10697,"threads_started":1}
I20260620 18:32:05.637398 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:05.621701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58778 (local address 127.10.131.3:37191)
0620 18:32:05.623689 (+ 1988us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:05.623700 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:05.629941 (+ 6241us) server_negotiation.cc:407] Connection header received
0620 18:32:05.630070 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:05.630080 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:05.630341 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:05.630553 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:05.631257 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:05.632435 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:05.634338 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:05.635718 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:05.635869 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:05.636092 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:05.636592 (+ 500us) server_negotiation.cc:299] Negotiation successful
0620 18:32:05.636751 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1756,"thread_start_us":387,"threads_started":1}
I20260620 18:32:05.637724 12163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1695dab04b194158b55ed646e33d0ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:05.644793 12302 tablet_service.cc:1467] Processing CreateTablet for tablet 1695dab04b194158b55ed646e33d0ee2 (DEFAULT_TABLE table=test-table [id=3cd4b4a2c3c541d889ac5bb0afae48a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:05.646270 12302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1695dab04b194158b55ed646e33d0ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:05.649832 12391 tablet_bootstrap.cc:492] T 1695dab04b194158b55ed646e33d0ee2 P 8d97b360ab544c8ba86d44fc332d7312: Bootstrap starting.
I20260620 18:32:05.650441 12392 tablet_bootstrap.cc:492] T 1695dab04b194158b55ed646e33d0ee2 P ab3b9330a0544503a5fd00fa7f77db11: Bootstrap starting.
I20260620 18:32:05.653764 12392 tablet_bootstrap.cc:654] T 1695dab04b194158b55ed646e33d0ee2 P ab3b9330a0544503a5fd00fa7f77db11: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:05.653764 12391 tablet_bootstrap.cc:654] T 1695dab04b194158b55ed646e33d0ee2 P 8d97b360ab544c8ba86d44fc332d7312: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:05.655032 12392 log.cc:824] T 1695dab04b194158b55ed646e33d0ee2 P ab3b9330a0544503a5fd00fa7f77db11: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:05.655032 12391 log.cc:824] T 1695dab04b194158b55ed646e33d0ee2 P 8d97b360ab544c8ba86d44fc332d7312: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:05.658262 12392 tablet_bootstrap.cc:492] T 1695dab04b194158b55ed646e33d0ee2 P ab3b9330a0544503a5fd00fa7f77db11: No bootstrap required, opened a new log
I20260620 18:32:05.658288 12391 tablet_bootstrap.cc:492] T 1695dab04b194158b55ed646e33d0ee2 P 8d97b360ab544c8ba86d44fc332d7312: No bootstrap required, opened a new log
I20260620 18:32:05.658565 12392 ts_tablet_manager.cc:1397] T 1695dab04b194158b55ed646e33d0ee2 P ab3b9330a0544503a5fd00fa7f77db11: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 18:32:05.658818 12391 ts_tablet_manager.cc:1397] T 1695dab04b194158b55ed646e33d0ee2 P 8d97b360ab544c8ba86d44fc332d7312: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 18:32:05.659130 12395 tablet_bootstrap.cc:492] T 1695dab04b194158b55ed646e33d0ee2 P 57e470bede444347bb8661831e02d6b2: Bootstrap starting.
I20260620 18:32:05.663033 12395 tablet_bootstrap.cc:654] T 1695dab04b194158b55ed646e33d0ee2 P 57e470bede444347bb8661831e02d6b2: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:05.664390 12395 log.cc:824] T 1695dab04b194158b55ed646e33d0ee2 P 57e470bede444347bb8661831e02d6b2: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:05.666821 12391 raft_consensus.cc:348] T 1695dab04b194158b55ed646e33d0ee2 P 8d97b360ab544c8ba86d44fc332d7312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } } peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } }
I20260620 18:32:05.667210 12391 raft_consensus.cc:374] T 1695dab04b194158b55ed646e33d0ee2 P 8d97b360ab544c8ba86d44fc332d7312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:05.667308 12391 raft_consensus.cc:729] T 1695dab04b194158b55ed646e33d0ee2 P 8d97b360ab544c8ba86d44fc332d7312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d97b360ab544c8ba86d44fc332d7312, State: Initialized, Role: FOLLOWER
I20260620 18:32:05.667832 12395 tablet_bootstrap.cc:492] T 1695dab04b194158b55ed646e33d0ee2 P 57e470bede444347bb8661831e02d6b2: No bootstrap required, opened a new log
I20260620 18:32:05.668130 12395 ts_tablet_manager.cc:1397] T 1695dab04b194158b55ed646e33d0ee2 P 57e470bede444347bb8661831e02d6b2: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260620 18:32:05.668031 12391 consensus_queue.cc:260] T 1695dab04b194158b55ed646e33d0ee2 P 8d97b360ab544c8ba86d44fc332d7312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } } peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } }
I20260620 18:32:05.669517 12392 raft_consensus.cc:348] T 1695dab04b194158b55ed646e33d0ee2 P ab3b9330a0544503a5fd00fa7f77db11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } } peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } }
I20260620 18:32:05.669955 12392 raft_consensus.cc:374] T 1695dab04b194158b55ed646e33d0ee2 P ab3b9330a0544503a5fd00fa7f77db11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:05.670053 12392 raft_consensus.cc:729] T 1695dab04b194158b55ed646e33d0ee2 P ab3b9330a0544503a5fd00fa7f77db11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3b9330a0544503a5fd00fa7f77db11, State: Initialized, Role: FOLLOWER
I20260620 18:32:05.670722 12392 consensus_queue.cc:260] T 1695dab04b194158b55ed646e33d0ee2 P ab3b9330a0544503a5fd00fa7f77db11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } } peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } }
I20260620 18:32:05.671043 12391 ts_tablet_manager.cc:1428] T 1695dab04b194158b55ed646e33d0ee2 P 8d97b360ab544c8ba86d44fc332d7312: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260620 18:32:05.671989 12233 heartbeater.cc:503] Master 127.10.131.62:45203 was elected leader, sending a full tablet report...
I20260620 18:32:05.672609 12392 ts_tablet_manager.cc:1428] T 1695dab04b194158b55ed646e33d0ee2 P ab3b9330a0544503a5fd00fa7f77db11: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260620 18:32:05.679052 12395 raft_consensus.cc:348] T 1695dab04b194158b55ed646e33d0ee2 P 57e470bede444347bb8661831e02d6b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } } peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } }
I20260620 18:32:05.679454 12395 raft_consensus.cc:374] T 1695dab04b194158b55ed646e33d0ee2 P 57e470bede444347bb8661831e02d6b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:05.679560 12395 raft_consensus.cc:729] T 1695dab04b194158b55ed646e33d0ee2 P 57e470bede444347bb8661831e02d6b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57e470bede444347bb8661831e02d6b2, State: Initialized, Role: FOLLOWER
I20260620 18:32:05.680096 12395 consensus_queue.cc:260] T 1695dab04b194158b55ed646e33d0ee2 P 57e470bede444347bb8661831e02d6b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } } peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } }
I20260620 18:32:05.681217 12372 heartbeater.cc:503] Master 127.10.131.62:45203 was elected leader, sending a full tablet report...
I20260620 18:32:05.681887 12395 ts_tablet_manager.cc:1428] T 1695dab04b194158b55ed646e33d0ee2 P 57e470bede444347bb8661831e02d6b2: Time spent starting tablet: real 0.014s user 0.012s sys 0.002s
I20260620 18:32:05.716297 12397 raft_consensus.cc:482] T 1695dab04b194158b55ed646e33d0ee2 P 8d97b360ab544c8ba86d44fc332d7312 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:32:05.716728 12397 raft_consensus.cc:504] T 1695dab04b194158b55ed646e33d0ee2 P 8d97b360ab544c8ba86d44fc332d7312 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } } peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } }
I20260620 18:32:05.718595 12397 leader_election.cc:288] T 1695dab04b194158b55ed646e33d0ee2 P 8d97b360ab544c8ba86d44fc332d7312 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab3b9330a0544503a5fd00fa7f77db11 (127.10.131.2:40403), 57e470bede444347bb8661831e02d6b2 (127.10.131.3:37191)
I20260620 18:32:05.727558 12386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:05.719009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:58479 (local address 127.10.131.2:40403)
0620 18:32:05.719182 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:05.719186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:05.719824 (+ 638us) server_negotiation.cc:407] Connection header received
0620 18:32:05.720115 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:05.720120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:05.720224 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:05.720325 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:05.721047 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:05.722070 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:05.725068 (+ 2998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:05.726311 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:05.726509 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:05.726590 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:05.727180 (+ 590us) server_negotiation.cc:299] Negotiation successful
0620 18:32:05.727310 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 18:32:05.727584 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:05.720099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:53225 (local address 127.10.131.3:37191)
0620 18:32:05.720271 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:05.720275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:05.720729 (+ 454us) server_negotiation.cc:407] Connection header received
0620 18:32:05.720893 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:05.720898 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:05.720957 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:05.721069 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:05.721572 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:05.722471 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:05.725062 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:05.726311 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:05.726506 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:05.726590 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:05.727180 (+ 590us) server_negotiation.cc:299] Negotiation successful
0620 18:32:05.727310 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 18:32:05.727756 12401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:05.719800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:37191 (local address 127.10.131.1:53225)
0620 18:32:05.720598 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:05.720615 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:05.720736 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:05.721079 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:05.721084 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:05.721135 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:05.721409 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:05.721417 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:05.722573 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:05.722578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:05.724877 (+ 2299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:05.724889 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:05.726670 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:05.726678 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:05.726944 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:05.727026 (+ 82us) client_negotiation.cc:769] Sending connection context
0620 18:32:05.727121 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 18:32:05.727296 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"mutex_wait_us":227,"thread_start_us":495,"threads_started":1}
I20260620 18:32:05.728226 12400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:05.719008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:40403 (local address 127.10.131.1:58479)
0620 18:32:05.719718 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:05.719734 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:05.719965 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:05.720375 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:05.720381 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:05.720416 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:05.720886 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:05.720900 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:05.722272 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:05.722278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:05.724875 (+ 2597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:05.724889 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:05.726687 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:05.726692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:05.726923 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:05.727024 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 18:32:05.727121 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 18:32:05.727296 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":153,"thread_start_us":412,"threads_started":1}
I20260620 18:32:05.728874 12322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1695dab04b194158b55ed646e33d0ee2" candidate_uuid: "8d97b360ab544c8ba86d44fc332d7312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57e470bede444347bb8661831e02d6b2" is_pre_election: true
I20260620 18:32:05.729281 12187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1695dab04b194158b55ed646e33d0ee2" candidate_uuid: "8d97b360ab544c8ba86d44fc332d7312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab3b9330a0544503a5fd00fa7f77db11" is_pre_election: true
I20260620 18:32:05.729530 12322 raft_consensus.cc:2454] T 1695dab04b194158b55ed646e33d0ee2 P 57e470bede444347bb8661831e02d6b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d97b360ab544c8ba86d44fc332d7312 in term 0.
I20260620 18:32:05.729800 12187 raft_consensus.cc:2454] T 1695dab04b194158b55ed646e33d0ee2 P ab3b9330a0544503a5fd00fa7f77db11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d97b360ab544c8ba86d44fc332d7312 in term 0.
I20260620 18:32:05.730298 11977 leader_election.cc:302] T 1695dab04b194158b55ed646e33d0ee2 P 8d97b360ab544c8ba86d44fc332d7312 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57e470bede444347bb8661831e02d6b2, 8d97b360ab544c8ba86d44fc332d7312; no voters:
I20260620 18:32:05.730690 12397 raft_consensus.cc:2781] T 1695dab04b194158b55ed646e33d0ee2 P 8d97b360ab544c8ba86d44fc332d7312 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:32:05.730804 12397 raft_consensus.cc:482] T 1695dab04b194158b55ed646e33d0ee2 P 8d97b360ab544c8ba86d44fc332d7312 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:32:05.730947 12397 raft_consensus.cc:3037] T 1695dab04b194158b55ed646e33d0ee2 P 8d97b360ab544c8ba86d44fc332d7312 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:05.733142 12397 raft_consensus.cc:504] T 1695dab04b194158b55ed646e33d0ee2 P 8d97b360ab544c8ba86d44fc332d7312 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } } peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } }
I20260620 18:32:05.733718 12397 leader_election.cc:288] T 1695dab04b194158b55ed646e33d0ee2 P 8d97b360ab544c8ba86d44fc332d7312 [CANDIDATE]: Term 1 election: Requested vote from peers ab3b9330a0544503a5fd00fa7f77db11 (127.10.131.2:40403), 57e470bede444347bb8661831e02d6b2 (127.10.131.3:37191)
I20260620 18:32:05.734197 12187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1695dab04b194158b55ed646e33d0ee2" candidate_uuid: "8d97b360ab544c8ba86d44fc332d7312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab3b9330a0544503a5fd00fa7f77db11"
I20260620 18:32:05.734304 12322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1695dab04b194158b55ed646e33d0ee2" candidate_uuid: "8d97b360ab544c8ba86d44fc332d7312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57e470bede444347bb8661831e02d6b2"
I20260620 18:32:05.734402 12187 raft_consensus.cc:3037] T 1695dab04b194158b55ed646e33d0ee2 P ab3b9330a0544503a5fd00fa7f77db11 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:05.734504 12322 raft_consensus.cc:3037] T 1695dab04b194158b55ed646e33d0ee2 P 57e470bede444347bb8661831e02d6b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:05.737622 12322 raft_consensus.cc:2454] T 1695dab04b194158b55ed646e33d0ee2 P 57e470bede444347bb8661831e02d6b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d97b360ab544c8ba86d44fc332d7312 in term 1.
I20260620 18:32:05.737622 12187 raft_consensus.cc:2454] T 1695dab04b194158b55ed646e33d0ee2 P ab3b9330a0544503a5fd00fa7f77db11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d97b360ab544c8ba86d44fc332d7312 in term 1.
I20260620 18:32:05.738140 11977 leader_election.cc:302] T 1695dab04b194158b55ed646e33d0ee2 P 8d97b360ab544c8ba86d44fc332d7312 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d97b360ab544c8ba86d44fc332d7312, ab3b9330a0544503a5fd00fa7f77db11; no voters:
I20260620 18:32:05.738476 12397 raft_consensus.cc:2781] T 1695dab04b194158b55ed646e33d0ee2 P 8d97b360ab544c8ba86d44fc332d7312 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:05.738859 12397 raft_consensus.cc:686] T 1695dab04b194158b55ed646e33d0ee2 P 8d97b360ab544c8ba86d44fc332d7312 [term 1 LEADER]: Becoming Leader. State: Replica: 8d97b360ab544c8ba86d44fc332d7312, State: Running, Role: LEADER
I20260620 18:32:05.739307 12397 consensus_queue.cc:237] T 1695dab04b194158b55ed646e33d0ee2 P 8d97b360ab544c8ba86d44fc332d7312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } } peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } }
I20260620 18:32:05.743994 11886 catalog_manager.cc:5495] T 1695dab04b194158b55ed646e33d0ee2 P 8d97b360ab544c8ba86d44fc332d7312 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d97b360ab544c8ba86d44fc332d7312 (127.10.131.1). New cstate: current_term: 1 leader_uuid: "8d97b360ab544c8ba86d44fc332d7312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } health_report { overall_health: UNKNOWN } } }
W20260620 18:32:05.750056 12095 tablet.cc:2369] T 1695dab04b194158b55ed646e33d0ee2 P 8d97b360ab544c8ba86d44fc332d7312: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:32:05.753933 12234 tablet.cc:2369] T 1695dab04b194158b55ed646e33d0ee2 P ab3b9330a0544503a5fd00fa7f77db11: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:32:05.789808 12387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:05.784487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52812 (local address 127.10.131.1:39489)
0620 18:32:05.784684 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:05.784688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:05.784911 (+ 223us) server_negotiation.cc:407] Connection header received
0620 18:32:05.785147 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:05.785151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:05.785228 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:05.785355 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:32:05.785695 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:05.786383 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:05.787650 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:05.788037 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:05.788129 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:05.788589 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:32:05.789134 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:32:05.789237 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:05.789440 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 18:32:05.789502 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 18:32:05.792174 12373 tablet.cc:2369] T 1695dab04b194158b55ed646e33d0ee2 P 57e470bede444347bb8661831e02d6b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:32:05.797912 12187 raft_consensus.cc:1261] T 1695dab04b194158b55ed646e33d0ee2 P ab3b9330a0544503a5fd00fa7f77db11 [term 1 FOLLOWER]: Refusing update from remote peer 8d97b360ab544c8ba86d44fc332d7312: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:05.797981 12322 raft_consensus.cc:1261] T 1695dab04b194158b55ed646e33d0ee2 P 57e470bede444347bb8661831e02d6b2 [term 1 FOLLOWER]: Refusing update from remote peer 8d97b360ab544c8ba86d44fc332d7312: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:05.798902 12397 consensus_queue.cc:1035] T 1695dab04b194158b55ed646e33d0ee2 P 8d97b360ab544c8ba86d44fc332d7312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:05.799206 12402 consensus_queue.cc:1035] T 1695dab04b194158b55ed646e33d0ee2 P 8d97b360ab544c8ba86d44fc332d7312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:05.828884 11887 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:32:05.829174 11887 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:32:05.832696 11887 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1cb409d78ff4d0d8c15e24ede3ff4bd: Sending DeleteTablet for 3 replicas of tablet 1695dab04b194158b55ed646e33d0ee2
I20260620 18:32:05.833890 12024 tablet_service.cc:1514] Processing DeleteTablet for tablet 1695dab04b194158b55ed646e33d0ee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:05 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:52798
I20260620 18:32:05.834056 12302 tablet_service.cc:1514] Processing DeleteTablet for tablet 1695dab04b194158b55ed646e33d0ee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:05 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:58778
I20260620 18:32:05.834228 12163 tablet_service.cc:1514] Processing DeleteTablet for tablet 1695dab04b194158b55ed646e33d0ee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:05 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:38174
I20260620 18:32:05.837044 11887 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:32:05.837846 11887 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:32:05.849438 12415 tablet_replica.cc:330] T 1695dab04b194158b55ed646e33d0ee2 P 8d97b360ab544c8ba86d44fc332d7312: stopping tablet replica
I20260620 18:32:05.849601 12301 tablet_service.cc:1467] Processing CreateTablet for tablet 74f3b50dba41448eb165cfb69bd470c7 (DEFAULT_TABLE table=test-table [id=7e7b8ab58b0c4d63a3edc3532cfa4062]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:05.850170 12301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74f3b50dba41448eb165cfb69bd470c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:05.850203 12415 raft_consensus.cc:2229] T 1695dab04b194158b55ed646e33d0ee2 P 8d97b360ab544c8ba86d44fc332d7312 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:32:05.851015 12415 raft_consensus.cc:2258] T 1695dab04b194158b55ed646e33d0ee2 P 8d97b360ab544c8ba86d44fc332d7312 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:05.853238 12415 ts_tablet_manager.cc:1905] T 1695dab04b194158b55ed646e33d0ee2 P 8d97b360ab544c8ba86d44fc332d7312: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:05.853709 12024 tablet_service.cc:1467] Processing CreateTablet for tablet 74f3b50dba41448eb165cfb69bd470c7 (DEFAULT_TABLE table=test-table [id=7e7b8ab58b0c4d63a3edc3532cfa4062]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:05.853789 12162 tablet_service.cc:1467] Processing CreateTablet for tablet 74f3b50dba41448eb165cfb69bd470c7 (DEFAULT_TABLE table=test-table [id=7e7b8ab58b0c4d63a3edc3532cfa4062]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:05.854101 12024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74f3b50dba41448eb165cfb69bd470c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:05.854517 12162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74f3b50dba41448eb165cfb69bd470c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:05.858225 12395 tablet_bootstrap.cc:492] T 74f3b50dba41448eb165cfb69bd470c7 P 57e470bede444347bb8661831e02d6b2: Bootstrap starting.
I20260620 18:32:05.860442 12395 tablet_bootstrap.cc:654] T 74f3b50dba41448eb165cfb69bd470c7 P 57e470bede444347bb8661831e02d6b2: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:05.865998 12416 tablet_replica.cc:330] T 1695dab04b194158b55ed646e33d0ee2 P 57e470bede444347bb8661831e02d6b2: stopping tablet replica
I20260620 18:32:05.866437 12416 raft_consensus.cc:2229] T 1695dab04b194158b55ed646e33d0ee2 P 57e470bede444347bb8661831e02d6b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:32:05.866923 12416 raft_consensus.cc:2258] T 1695dab04b194158b55ed646e33d0ee2 P 57e470bede444347bb8661831e02d6b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:05.867257 12392 tablet_bootstrap.cc:492] T 74f3b50dba41448eb165cfb69bd470c7 P ab3b9330a0544503a5fd00fa7f77db11: Bootstrap starting.
I20260620 18:32:05.867928 12391 tablet_bootstrap.cc:492] T 74f3b50dba41448eb165cfb69bd470c7 P 8d97b360ab544c8ba86d44fc332d7312: Bootstrap starting.
I20260620 18:32:05.868912 12392 tablet_bootstrap.cc:654] T 74f3b50dba41448eb165cfb69bd470c7 P ab3b9330a0544503a5fd00fa7f77db11: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:05.869580 12391 tablet_bootstrap.cc:654] T 74f3b50dba41448eb165cfb69bd470c7 P 8d97b360ab544c8ba86d44fc332d7312: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:05.875942 12395 tablet_bootstrap.cc:492] T 74f3b50dba41448eb165cfb69bd470c7 P 57e470bede444347bb8661831e02d6b2: No bootstrap required, opened a new log
I20260620 18:32:05.876132 12395 ts_tablet_manager.cc:1397] T 74f3b50dba41448eb165cfb69bd470c7 P 57e470bede444347bb8661831e02d6b2: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20260620 18:32:05.877014 12395 raft_consensus.cc:348] T 74f3b50dba41448eb165cfb69bd470c7 P 57e470bede444347bb8661831e02d6b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } } peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } }
I20260620 18:32:05.877266 12395 raft_consensus.cc:374] T 74f3b50dba41448eb165cfb69bd470c7 P 57e470bede444347bb8661831e02d6b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:05.877372 12395 raft_consensus.cc:729] T 74f3b50dba41448eb165cfb69bd470c7 P 57e470bede444347bb8661831e02d6b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57e470bede444347bb8661831e02d6b2, State: Initialized, Role: FOLLOWER
I20260620 18:32:05.877629 12415 ts_tablet_manager.cc:1918] T 1695dab04b194158b55ed646e33d0ee2 P 8d97b360ab544c8ba86d44fc332d7312: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:32:05.877637 12395 consensus_queue.cc:260] T 74f3b50dba41448eb165cfb69bd470c7 P 57e470bede444347bb8661831e02d6b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } } peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } }
I20260620 18:32:05.878221 12395 ts_tablet_manager.cc:1428] T 74f3b50dba41448eb165cfb69bd470c7 P 57e470bede444347bb8661831e02d6b2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:32:05.880740 12417 tablet_replica.cc:330] T 1695dab04b194158b55ed646e33d0ee2 P ab3b9330a0544503a5fd00fa7f77db11: stopping tablet replica
I20260620 18:32:05.881168 12417 raft_consensus.cc:2229] T 1695dab04b194158b55ed646e33d0ee2 P ab3b9330a0544503a5fd00fa7f77db11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:32:05.882211 12392 tablet_bootstrap.cc:492] T 74f3b50dba41448eb165cfb69bd470c7 P ab3b9330a0544503a5fd00fa7f77db11: No bootstrap required, opened a new log
I20260620 18:32:05.882324 12392 ts_tablet_manager.cc:1397] T 74f3b50dba41448eb165cfb69bd470c7 P ab3b9330a0544503a5fd00fa7f77db11: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.004s
I20260620 18:32:05.883003 12392 raft_consensus.cc:348] T 74f3b50dba41448eb165cfb69bd470c7 P ab3b9330a0544503a5fd00fa7f77db11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } } peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } }
I20260620 18:32:05.883196 12392 raft_consensus.cc:374] T 74f3b50dba41448eb165cfb69bd470c7 P ab3b9330a0544503a5fd00fa7f77db11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:05.883266 12392 raft_consensus.cc:729] T 74f3b50dba41448eb165cfb69bd470c7 P ab3b9330a0544503a5fd00fa7f77db11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3b9330a0544503a5fd00fa7f77db11, State: Initialized, Role: FOLLOWER
I20260620 18:32:05.883533 12392 consensus_queue.cc:260] T 74f3b50dba41448eb165cfb69bd470c7 P ab3b9330a0544503a5fd00fa7f77db11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } } peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } }
I20260620 18:32:05.886420 12392 ts_tablet_manager.cc:1428] T 74f3b50dba41448eb165cfb69bd470c7 P ab3b9330a0544503a5fd00fa7f77db11: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 18:32:05.888743 12417 raft_consensus.cc:2258] T 1695dab04b194158b55ed646e33d0ee2 P ab3b9330a0544503a5fd00fa7f77db11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:05.890869 12417 ts_tablet_manager.cc:1905] T 1695dab04b194158b55ed646e33d0ee2 P ab3b9330a0544503a5fd00fa7f77db11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:05.891367 12416 ts_tablet_manager.cc:1905] T 1695dab04b194158b55ed646e33d0ee2 P 57e470bede444347bb8661831e02d6b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:05.897787 12417 ts_tablet_manager.cc:1918] T 1695dab04b194158b55ed646e33d0ee2 P ab3b9330a0544503a5fd00fa7f77db11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:32:05.897933 12417 log.cc:1196] T 1695dab04b194158b55ed646e33d0ee2 P ab3b9330a0544503a5fd00fa7f77db11: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-1/wal/wals/1695dab04b194158b55ed646e33d0ee2
I20260620 18:32:05.897945 12416 ts_tablet_manager.cc:1918] T 1695dab04b194158b55ed646e33d0ee2 P 57e470bede444347bb8661831e02d6b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:32:05.898093 12416 log.cc:1196] T 1695dab04b194158b55ed646e33d0ee2 P 57e470bede444347bb8661831e02d6b2: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-2/wal/wals/1695dab04b194158b55ed646e33d0ee2
I20260620 18:32:05.898470 12417 ts_tablet_manager.cc:1939] T 1695dab04b194158b55ed646e33d0ee2 P ab3b9330a0544503a5fd00fa7f77db11: Deleting consensus metadata
I20260620 18:32:05.898597 12416 ts_tablet_manager.cc:1939] T 1695dab04b194158b55ed646e33d0ee2 P 57e470bede444347bb8661831e02d6b2: Deleting consensus metadata
I20260620 18:32:05.897927 12415 log.cc:1196] T 1695dab04b194158b55ed646e33d0ee2 P 8d97b360ab544c8ba86d44fc332d7312: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-0/wal/wals/1695dab04b194158b55ed646e33d0ee2
I20260620 18:32:05.899968 12415 ts_tablet_manager.cc:1939] T 1695dab04b194158b55ed646e33d0ee2 P 8d97b360ab544c8ba86d44fc332d7312: Deleting consensus metadata
I20260620 18:32:05.900550 11874 catalog_manager.cc:4841] TS ab3b9330a0544503a5fd00fa7f77db11 (127.10.131.2:40403): tablet 1695dab04b194158b55ed646e33d0ee2 (table test-table [id=3cd4b4a2c3c541d889ac5bb0afae48a5]) successfully deleted
I20260620 18:32:05.900890 11874 catalog_manager.cc:4841] TS 57e470bede444347bb8661831e02d6b2 (127.10.131.3:37191): tablet 1695dab04b194158b55ed646e33d0ee2 (table test-table [id=3cd4b4a2c3c541d889ac5bb0afae48a5]) successfully deleted
I20260620 18:32:05.901975 12391 tablet_bootstrap.cc:492] T 74f3b50dba41448eb165cfb69bd470c7 P 8d97b360ab544c8ba86d44fc332d7312: No bootstrap required, opened a new log
I20260620 18:32:05.902146 12391 ts_tablet_manager.cc:1397] T 74f3b50dba41448eb165cfb69bd470c7 P 8d97b360ab544c8ba86d44fc332d7312: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.013s
I20260620 18:32:05.902809 11871 catalog_manager.cc:4841] TS 8d97b360ab544c8ba86d44fc332d7312 (127.10.131.1:39489): tablet 1695dab04b194158b55ed646e33d0ee2 (table test-table [id=3cd4b4a2c3c541d889ac5bb0afae48a5]) successfully deleted
I20260620 18:32:05.903060 12391 raft_consensus.cc:348] T 74f3b50dba41448eb165cfb69bd470c7 P 8d97b360ab544c8ba86d44fc332d7312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } } peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } }
I20260620 18:32:05.903311 12391 raft_consensus.cc:374] T 74f3b50dba41448eb165cfb69bd470c7 P 8d97b360ab544c8ba86d44fc332d7312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:05.903417 12391 raft_consensus.cc:729] T 74f3b50dba41448eb165cfb69bd470c7 P 8d97b360ab544c8ba86d44fc332d7312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d97b360ab544c8ba86d44fc332d7312, State: Initialized, Role: FOLLOWER
I20260620 18:32:05.903697 12391 consensus_queue.cc:260] T 74f3b50dba41448eb165cfb69bd470c7 P 8d97b360ab544c8ba86d44fc332d7312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } } peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } }
I20260620 18:32:05.904206 12391 ts_tablet_manager.cc:1428] T 74f3b50dba41448eb165cfb69bd470c7 P 8d97b360ab544c8ba86d44fc332d7312: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 18:32:06.179675 12402 raft_consensus.cc:482] T 74f3b50dba41448eb165cfb69bd470c7 P 8d97b360ab544c8ba86d44fc332d7312 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:32:06.179942 12402 raft_consensus.cc:504] T 74f3b50dba41448eb165cfb69bd470c7 P 8d97b360ab544c8ba86d44fc332d7312 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } } peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } }
I20260620 18:32:06.180657 12402 leader_election.cc:288] T 74f3b50dba41448eb165cfb69bd470c7 P 8d97b360ab544c8ba86d44fc332d7312 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab3b9330a0544503a5fd00fa7f77db11 (127.10.131.2:40403), 57e470bede444347bb8661831e02d6b2 (127.10.131.3:37191)
I20260620 18:32:06.181273 12187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f3b50dba41448eb165cfb69bd470c7" candidate_uuid: "8d97b360ab544c8ba86d44fc332d7312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab3b9330a0544503a5fd00fa7f77db11" is_pre_election: true
I20260620 18:32:06.181324 12322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f3b50dba41448eb165cfb69bd470c7" candidate_uuid: "8d97b360ab544c8ba86d44fc332d7312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57e470bede444347bb8661831e02d6b2" is_pre_election: true
I20260620 18:32:06.181514 12187 raft_consensus.cc:2454] T 74f3b50dba41448eb165cfb69bd470c7 P ab3b9330a0544503a5fd00fa7f77db11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d97b360ab544c8ba86d44fc332d7312 in term 0.
I20260620 18:32:06.181602 12322 raft_consensus.cc:2454] T 74f3b50dba41448eb165cfb69bd470c7 P 57e470bede444347bb8661831e02d6b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d97b360ab544c8ba86d44fc332d7312 in term 0.
I20260620 18:32:06.182029 11977 leader_election.cc:302] T 74f3b50dba41448eb165cfb69bd470c7 P 8d97b360ab544c8ba86d44fc332d7312 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d97b360ab544c8ba86d44fc332d7312, ab3b9330a0544503a5fd00fa7f77db11; no voters:
I20260620 18:32:06.182359 12402 raft_consensus.cc:2781] T 74f3b50dba41448eb165cfb69bd470c7 P 8d97b360ab544c8ba86d44fc332d7312 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:32:06.182494 12402 raft_consensus.cc:482] T 74f3b50dba41448eb165cfb69bd470c7 P 8d97b360ab544c8ba86d44fc332d7312 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:32:06.182633 12402 raft_consensus.cc:3037] T 74f3b50dba41448eb165cfb69bd470c7 P 8d97b360ab544c8ba86d44fc332d7312 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:06.184809 12402 raft_consensus.cc:504] T 74f3b50dba41448eb165cfb69bd470c7 P 8d97b360ab544c8ba86d44fc332d7312 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } } peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } }
I20260620 18:32:06.185406 12402 leader_election.cc:288] T 74f3b50dba41448eb165cfb69bd470c7 P 8d97b360ab544c8ba86d44fc332d7312 [CANDIDATE]: Term 1 election: Requested vote from peers ab3b9330a0544503a5fd00fa7f77db11 (127.10.131.2:40403), 57e470bede444347bb8661831e02d6b2 (127.10.131.3:37191)
I20260620 18:32:06.185931 12187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f3b50dba41448eb165cfb69bd470c7" candidate_uuid: "8d97b360ab544c8ba86d44fc332d7312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab3b9330a0544503a5fd00fa7f77db11"
I20260620 18:32:06.185969 12322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f3b50dba41448eb165cfb69bd470c7" candidate_uuid: "8d97b360ab544c8ba86d44fc332d7312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57e470bede444347bb8661831e02d6b2"
I20260620 18:32:06.186131 12187 raft_consensus.cc:3037] T 74f3b50dba41448eb165cfb69bd470c7 P ab3b9330a0544503a5fd00fa7f77db11 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:06.186195 12322 raft_consensus.cc:3037] T 74f3b50dba41448eb165cfb69bd470c7 P 57e470bede444347bb8661831e02d6b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:06.188031 12187 raft_consensus.cc:2454] T 74f3b50dba41448eb165cfb69bd470c7 P ab3b9330a0544503a5fd00fa7f77db11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d97b360ab544c8ba86d44fc332d7312 in term 1.
I20260620 18:32:06.188407 11977 leader_election.cc:302] T 74f3b50dba41448eb165cfb69bd470c7 P 8d97b360ab544c8ba86d44fc332d7312 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d97b360ab544c8ba86d44fc332d7312, ab3b9330a0544503a5fd00fa7f77db11; no voters:
I20260620 18:32:06.188675 12402 raft_consensus.cc:2781] T 74f3b50dba41448eb165cfb69bd470c7 P 8d97b360ab544c8ba86d44fc332d7312 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:06.188913 12402 raft_consensus.cc:686] T 74f3b50dba41448eb165cfb69bd470c7 P 8d97b360ab544c8ba86d44fc332d7312 [term 1 LEADER]: Becoming Leader. State: Replica: 8d97b360ab544c8ba86d44fc332d7312, State: Running, Role: LEADER
I20260620 18:32:06.189216 12402 consensus_queue.cc:237] T 74f3b50dba41448eb165cfb69bd470c7 P 8d97b360ab544c8ba86d44fc332d7312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } } peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } }
I20260620 18:32:06.189512 12322 raft_consensus.cc:2454] T 74f3b50dba41448eb165cfb69bd470c7 P 57e470bede444347bb8661831e02d6b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d97b360ab544c8ba86d44fc332d7312 in term 1.
I20260620 18:32:06.192118 11886 catalog_manager.cc:5495] T 74f3b50dba41448eb165cfb69bd470c7 P 8d97b360ab544c8ba86d44fc332d7312 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d97b360ab544c8ba86d44fc332d7312 (127.10.131.1). New cstate: current_term: 1 leader_uuid: "8d97b360ab544c8ba86d44fc332d7312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } health_report { overall_health: UNKNOWN } } }
Jun 20 18:32:06 dist-test-slave-cbn3 krb5kdc[11842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980323, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:32:06.271967 12428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:06.255660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:45203 (local address 127.0.0.1:55958)
0620 18:32:06.256567 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:06.256585 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:06.256711 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:06.257135 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:06.257138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:06.257787 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:06.258019 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:06.258038 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:06.259593 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:06.259596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:06.260549 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:06.260562 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:06.261674 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:06.261679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:06.261851 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:06.262582 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:06.262599 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:06.264774 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:06.268806 (+ 4032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:06.268821 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:06.268829 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:06.269445 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:06.269843 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:06.269865 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:06.269869 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:06.270090 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:06.270870 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:06.270879 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:06.271125 (+ 246us) client_negotiation.cc:769] Sending connection context
0620 18:32:06.271348 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 18:32:06.271632 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"mutex_wait_us":33,"thread_start_us":621,"threads_started":1}
I20260620 18:32:06.272058 12427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:06.255627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55958 (local address 127.10.131.62:45203)
0620 18:32:06.256386 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:06.256396 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:06.256680 (+ 284us) server_negotiation.cc:407] Connection header received
0620 18:32:06.256833 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:06.256858 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:06.256956 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:06.257099 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:06.258210 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:06.259425 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:06.260697 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:06.261497 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:06.261657 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:06.265031 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:06.265039 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:06.265044 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:06.265083 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:06.268642 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:06.269576 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:06.269580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:06.269584 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:06.269711 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:06.270239 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:06.270244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:06.270247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:06.270693 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:06.270845 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:06.271440 (+ 595us) server_negotiation.cc:299] Negotiation successful
0620 18:32:06.271738 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":567,"thread_start_us":350,"threads_started":1}
I20260620 18:32:06.277017 11886 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:55958:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 18:32:06.278132 11886 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:32:06.284670 12024 tablet_service.cc:1467] Processing CreateTablet for tablet 35ec79b620cd43728bd9b74496eeeefc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d1c88c679f5e4164b32d3fa3cfe4700c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:32:06.285296 12024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35ec79b620cd43728bd9b74496eeeefc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:06.285262 12302 tablet_service.cc:1467] Processing CreateTablet for tablet 35ec79b620cd43728bd9b74496eeeefc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d1c88c679f5e4164b32d3fa3cfe4700c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:32:06.285338 12163 tablet_service.cc:1467] Processing CreateTablet for tablet 35ec79b620cd43728bd9b74496eeeefc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d1c88c679f5e4164b32d3fa3cfe4700c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:32:06.285756 12302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35ec79b620cd43728bd9b74496eeeefc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:06.285974 12163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35ec79b620cd43728bd9b74496eeeefc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:06.292701 12395 tablet_bootstrap.cc:492] T 35ec79b620cd43728bd9b74496eeeefc P 57e470bede444347bb8661831e02d6b2: Bootstrap starting.
I20260620 18:32:06.292716 12392 tablet_bootstrap.cc:492] T 35ec79b620cd43728bd9b74496eeeefc P ab3b9330a0544503a5fd00fa7f77db11: Bootstrap starting.
I20260620 18:32:06.292729 12391 tablet_bootstrap.cc:492] T 35ec79b620cd43728bd9b74496eeeefc P 8d97b360ab544c8ba86d44fc332d7312: Bootstrap starting.
I20260620 18:32:06.294327 12395 tablet_bootstrap.cc:654] T 35ec79b620cd43728bd9b74496eeeefc P 57e470bede444347bb8661831e02d6b2: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:06.294574 12392 tablet_bootstrap.cc:654] T 35ec79b620cd43728bd9b74496eeeefc P ab3b9330a0544503a5fd00fa7f77db11: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:06.294574 12391 tablet_bootstrap.cc:654] T 35ec79b620cd43728bd9b74496eeeefc P 8d97b360ab544c8ba86d44fc332d7312: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:06.296404 12395 tablet_bootstrap.cc:492] T 35ec79b620cd43728bd9b74496eeeefc P 57e470bede444347bb8661831e02d6b2: No bootstrap required, opened a new log
I20260620 18:32:06.296586 12395 ts_tablet_manager.cc:1397] T 35ec79b620cd43728bd9b74496eeeefc P 57e470bede444347bb8661831e02d6b2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 18:32:06.297226 12395 raft_consensus.cc:348] T 35ec79b620cd43728bd9b74496eeeefc P 57e470bede444347bb8661831e02d6b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } } peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } }
I20260620 18:32:06.297343 12392 tablet_bootstrap.cc:492] T 35ec79b620cd43728bd9b74496eeeefc P ab3b9330a0544503a5fd00fa7f77db11: No bootstrap required, opened a new log
I20260620 18:32:06.297438 12395 raft_consensus.cc:374] T 35ec79b620cd43728bd9b74496eeeefc P 57e470bede444347bb8661831e02d6b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:06.297521 12395 raft_consensus.cc:729] T 35ec79b620cd43728bd9b74496eeeefc P 57e470bede444347bb8661831e02d6b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57e470bede444347bb8661831e02d6b2, State: Initialized, Role: FOLLOWER
I20260620 18:32:06.297537 12392 ts_tablet_manager.cc:1397] T 35ec79b620cd43728bd9b74496eeeefc P ab3b9330a0544503a5fd00fa7f77db11: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 18:32:06.297749 12395 consensus_queue.cc:260] T 35ec79b620cd43728bd9b74496eeeefc P 57e470bede444347bb8661831e02d6b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } } peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } }
I20260620 18:32:06.298231 12395 ts_tablet_manager.cc:1428] T 35ec79b620cd43728bd9b74496eeeefc P 57e470bede444347bb8661831e02d6b2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:32:06.298454 12392 raft_consensus.cc:348] T 35ec79b620cd43728bd9b74496eeeefc P ab3b9330a0544503a5fd00fa7f77db11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } } peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } }
I20260620 18:32:06.298574 12399 tablet_replica.cc:439] T 35ec79b620cd43728bd9b74496eeeefc P 57e470bede444347bb8661831e02d6b2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } } peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } } }
I20260620 18:32:06.298763 12392 raft_consensus.cc:374] T 35ec79b620cd43728bd9b74496eeeefc P ab3b9330a0544503a5fd00fa7f77db11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:06.298861 12399 tablet_replica.cc:442] T 35ec79b620cd43728bd9b74496eeeefc P 57e470bede444347bb8661831e02d6b2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:32:06.298844 12392 raft_consensus.cc:729] T 35ec79b620cd43728bd9b74496eeeefc P ab3b9330a0544503a5fd00fa7f77db11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3b9330a0544503a5fd00fa7f77db11, State: Initialized, Role: FOLLOWER
I20260620 18:32:06.299166 12392 consensus_queue.cc:260] T 35ec79b620cd43728bd9b74496eeeefc P ab3b9330a0544503a5fd00fa7f77db11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } } peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } }
I20260620 18:32:06.299744 12398 tablet_replica.cc:439] T 35ec79b620cd43728bd9b74496eeeefc P ab3b9330a0544503a5fd00fa7f77db11: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } } peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } } }
I20260620 18:32:06.300053 12398 tablet_replica.cc:442] T 35ec79b620cd43728bd9b74496eeeefc P ab3b9330a0544503a5fd00fa7f77db11: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:32:06.300680 12392 ts_tablet_manager.cc:1428] T 35ec79b620cd43728bd9b74496eeeefc P ab3b9330a0544503a5fd00fa7f77db11: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 18:32:06.300884 12391 tablet_bootstrap.cc:492] T 35ec79b620cd43728bd9b74496eeeefc P 8d97b360ab544c8ba86d44fc332d7312: No bootstrap required, opened a new log
I20260620 18:32:06.301074 12391 ts_tablet_manager.cc:1397] T 35ec79b620cd43728bd9b74496eeeefc P 8d97b360ab544c8ba86d44fc332d7312: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20260620 18:32:06.301699 12391 raft_consensus.cc:348] T 35ec79b620cd43728bd9b74496eeeefc P 8d97b360ab544c8ba86d44fc332d7312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } } peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } }
I20260620 18:32:06.302004 12391 raft_consensus.cc:374] T 35ec79b620cd43728bd9b74496eeeefc P 8d97b360ab544c8ba86d44fc332d7312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:06.302093 12391 raft_consensus.cc:729] T 35ec79b620cd43728bd9b74496eeeefc P 8d97b360ab544c8ba86d44fc332d7312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d97b360ab544c8ba86d44fc332d7312, State: Initialized, Role: FOLLOWER
I20260620 18:32:06.302336 12391 consensus_queue.cc:260] T 35ec79b620cd43728bd9b74496eeeefc P 8d97b360ab544c8ba86d44fc332d7312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } } peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } }
I20260620 18:32:06.302778 12402 tablet_replica.cc:439] T 35ec79b620cd43728bd9b74496eeeefc P 8d97b360ab544c8ba86d44fc332d7312: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } } peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } } }
I20260620 18:32:06.302873 12391 ts_tablet_manager.cc:1428] T 35ec79b620cd43728bd9b74496eeeefc P 8d97b360ab544c8ba86d44fc332d7312: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 18:32:06.303045 12402 tablet_replica.cc:442] T 35ec79b620cd43728bd9b74496eeeefc P 8d97b360ab544c8ba86d44fc332d7312: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:32:06.316692 12402 raft_consensus.cc:482] T 35ec79b620cd43728bd9b74496eeeefc P 8d97b360ab544c8ba86d44fc332d7312 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:32:06.316931 12402 raft_consensus.cc:504] T 35ec79b620cd43728bd9b74496eeeefc P 8d97b360ab544c8ba86d44fc332d7312 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } } peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } }
I20260620 18:32:06.317675 12402 leader_election.cc:288] T 35ec79b620cd43728bd9b74496eeeefc P 8d97b360ab544c8ba86d44fc332d7312 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab3b9330a0544503a5fd00fa7f77db11 (127.10.131.2:40403), 57e470bede444347bb8661831e02d6b2 (127.10.131.3:37191)
I20260620 18:32:06.318126 12187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35ec79b620cd43728bd9b74496eeeefc" candidate_uuid: "8d97b360ab544c8ba86d44fc332d7312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab3b9330a0544503a5fd00fa7f77db11" is_pre_election: true
I20260620 18:32:06.318188 12322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35ec79b620cd43728bd9b74496eeeefc" candidate_uuid: "8d97b360ab544c8ba86d44fc332d7312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57e470bede444347bb8661831e02d6b2" is_pre_election: true
I20260620 18:32:06.318397 12187 raft_consensus.cc:2454] T 35ec79b620cd43728bd9b74496eeeefc P ab3b9330a0544503a5fd00fa7f77db11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d97b360ab544c8ba86d44fc332d7312 in term 0.
I20260620 18:32:06.318446 12322 raft_consensus.cc:2454] T 35ec79b620cd43728bd9b74496eeeefc P 57e470bede444347bb8661831e02d6b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d97b360ab544c8ba86d44fc332d7312 in term 0.
I20260620 18:32:06.318831 11977 leader_election.cc:302] T 35ec79b620cd43728bd9b74496eeeefc P 8d97b360ab544c8ba86d44fc332d7312 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57e470bede444347bb8661831e02d6b2, 8d97b360ab544c8ba86d44fc332d7312; no voters:
I20260620 18:32:06.319079 12402 raft_consensus.cc:2781] T 35ec79b620cd43728bd9b74496eeeefc P 8d97b360ab544c8ba86d44fc332d7312 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:32:06.319207 12402 raft_consensus.cc:482] T 35ec79b620cd43728bd9b74496eeeefc P 8d97b360ab544c8ba86d44fc332d7312 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:32:06.319326 12402 raft_consensus.cc:3037] T 35ec79b620cd43728bd9b74496eeeefc P 8d97b360ab544c8ba86d44fc332d7312 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:06.321542 12402 raft_consensus.cc:504] T 35ec79b620cd43728bd9b74496eeeefc P 8d97b360ab544c8ba86d44fc332d7312 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } } peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } }
I20260620 18:32:06.322078 12402 leader_election.cc:288] T 35ec79b620cd43728bd9b74496eeeefc P 8d97b360ab544c8ba86d44fc332d7312 [CANDIDATE]: Term 1 election: Requested vote from peers ab3b9330a0544503a5fd00fa7f77db11 (127.10.131.2:40403), 57e470bede444347bb8661831e02d6b2 (127.10.131.3:37191)
I20260620 18:32:06.322619 12322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35ec79b620cd43728bd9b74496eeeefc" candidate_uuid: "8d97b360ab544c8ba86d44fc332d7312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57e470bede444347bb8661831e02d6b2"
I20260620 18:32:06.322841 12322 raft_consensus.cc:3037] T 35ec79b620cd43728bd9b74496eeeefc P 57e470bede444347bb8661831e02d6b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:06.324771 12322 raft_consensus.cc:2454] T 35ec79b620cd43728bd9b74496eeeefc P 57e470bede444347bb8661831e02d6b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d97b360ab544c8ba86d44fc332d7312 in term 1.
I20260620 18:32:06.325158 12187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35ec79b620cd43728bd9b74496eeeefc" candidate_uuid: "8d97b360ab544c8ba86d44fc332d7312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab3b9330a0544503a5fd00fa7f77db11"
I20260620 18:32:06.325229 11977 leader_election.cc:302] T 35ec79b620cd43728bd9b74496eeeefc P 8d97b360ab544c8ba86d44fc332d7312 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57e470bede444347bb8661831e02d6b2, 8d97b360ab544c8ba86d44fc332d7312; no voters:
I20260620 18:32:06.325390 12187 raft_consensus.cc:3037] T 35ec79b620cd43728bd9b74496eeeefc P ab3b9330a0544503a5fd00fa7f77db11 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:06.325517 12402 raft_consensus.cc:2781] T 35ec79b620cd43728bd9b74496eeeefc P 8d97b360ab544c8ba86d44fc332d7312 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:06.325690 12402 raft_consensus.cc:686] T 35ec79b620cd43728bd9b74496eeeefc P 8d97b360ab544c8ba86d44fc332d7312 [term 1 LEADER]: Becoming Leader. State: Replica: 8d97b360ab544c8ba86d44fc332d7312, State: Running, Role: LEADER
I20260620 18:32:06.325951 12402 consensus_queue.cc:237] T 35ec79b620cd43728bd9b74496eeeefc P 8d97b360ab544c8ba86d44fc332d7312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } } peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } }
I20260620 18:32:06.327629 12410 tablet_replica.cc:439] T 35ec79b620cd43728bd9b74496eeeefc P 8d97b360ab544c8ba86d44fc332d7312: TxnStatusTablet state changed. Reason: New leader 8d97b360ab544c8ba86d44fc332d7312. Latest consensus state: current_term: 1 leader_uuid: "8d97b360ab544c8ba86d44fc332d7312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } } peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } } }
I20260620 18:32:06.327921 12410 tablet_replica.cc:442] T 35ec79b620cd43728bd9b74496eeeefc P 8d97b360ab544c8ba86d44fc332d7312: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:32:06.328035 12187 raft_consensus.cc:2454] T 35ec79b620cd43728bd9b74496eeeefc P ab3b9330a0544503a5fd00fa7f77db11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d97b360ab544c8ba86d44fc332d7312 in term 1.
I20260620 18:32:06.329156 11886 catalog_manager.cc:5495] T 35ec79b620cd43728bd9b74496eeeefc P 8d97b360ab544c8ba86d44fc332d7312 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d97b360ab544c8ba86d44fc332d7312 (127.10.131.1). New cstate: current_term: 1 leader_uuid: "8d97b360ab544c8ba86d44fc332d7312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } health_report { overall_health: UNKNOWN } } }
I20260620 18:32:06.333216 12433 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 18:32:06.333429 12433 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 18:32:06.353657 12428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:06.347464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:39489 (local address 127.0.0.1:52816)
0620 18:32:06.347682 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:06.347710 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:06.347819 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:06.348579 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:06.348585 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:06.348600 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:32:06.348785 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:06.348796 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:06.349920 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:06.349926 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:06.351562 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:06.351577 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:06.352319 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:06.352326 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:06.352526 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:06.352597 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:32:06.353064 (+ 467us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:32:06.353087 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 18:32:06.353179 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 18:32:06.353340 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 18:32:06.353762 12435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:06.347477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52816 (local address 127.10.131.1:39489)
0620 18:32:06.348278 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:06.348283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:06.348300 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:32:06.348357 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:06.348360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:06.348424 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:06.348503 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:32:06.348983 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:06.349775 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:06.351704 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:06.352160 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:06.352269 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:06.352781 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:32:06.352951 (+ 170us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:32:06.353020 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:06.353378 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 18:32:06.353459 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":544,"threads_started":1}
I20260620 18:32:06.357897 12187 raft_consensus.cc:1261] T 35ec79b620cd43728bd9b74496eeeefc P ab3b9330a0544503a5fd00fa7f77db11 [term 1 FOLLOWER]: Refusing update from remote peer 8d97b360ab544c8ba86d44fc332d7312: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:06.357941 12322 raft_consensus.cc:1261] T 35ec79b620cd43728bd9b74496eeeefc P 57e470bede444347bb8661831e02d6b2 [term 1 FOLLOWER]: Refusing update from remote peer 8d97b360ab544c8ba86d44fc332d7312: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:06.358613 12410 consensus_queue.cc:1035] T 35ec79b620cd43728bd9b74496eeeefc P 8d97b360ab544c8ba86d44fc332d7312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:06.358899 12402 consensus_queue.cc:1035] T 35ec79b620cd43728bd9b74496eeeefc P 8d97b360ab544c8ba86d44fc332d7312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:06.366291 12399 tablet_replica.cc:439] T 35ec79b620cd43728bd9b74496eeeefc P 57e470bede444347bb8661831e02d6b2: TxnStatusTablet state changed. Reason: New leader 8d97b360ab544c8ba86d44fc332d7312. Latest consensus state: current_term: 1 leader_uuid: "8d97b360ab544c8ba86d44fc332d7312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } } peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } } }
I20260620 18:32:06.366587 12399 tablet_replica.cc:442] T 35ec79b620cd43728bd9b74496eeeefc P 57e470bede444347bb8661831e02d6b2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:32:06.371565 12399 tablet_replica.cc:439] T 35ec79b620cd43728bd9b74496eeeefc P 57e470bede444347bb8661831e02d6b2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8d97b360ab544c8ba86d44fc332d7312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } } peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } } }
I20260620 18:32:06.371857 12399 tablet_replica.cc:442] T 35ec79b620cd43728bd9b74496eeeefc P 57e470bede444347bb8661831e02d6b2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:32:06.370618 12436 mvcc.cc:205] Tried to move back new op lower bound from 7298991416755453952 to 7298991416633487360. Current Snapshot: MvccSnapshot[applied={T|T < 7298991416755453952}]
I20260620 18:32:06.373346 12402 tablet_replica.cc:439] T 35ec79b620cd43728bd9b74496eeeefc P 8d97b360ab544c8ba86d44fc332d7312: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8d97b360ab544c8ba86d44fc332d7312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } } peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } } }
I20260620 18:32:06.373656 12402 tablet_replica.cc:442] T 35ec79b620cd43728bd9b74496eeeefc P 8d97b360ab544c8ba86d44fc332d7312: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:32:06.376447 12398 tablet_replica.cc:439] T 35ec79b620cd43728bd9b74496eeeefc P ab3b9330a0544503a5fd00fa7f77db11: TxnStatusTablet state changed. Reason: New leader 8d97b360ab544c8ba86d44fc332d7312. Latest consensus state: current_term: 1 leader_uuid: "8d97b360ab544c8ba86d44fc332d7312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } } peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } } }
I20260620 18:32:06.376747 12398 tablet_replica.cc:442] T 35ec79b620cd43728bd9b74496eeeefc P ab3b9330a0544503a5fd00fa7f77db11: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:32:06.376683 12402 tablet_replica.cc:439] T 35ec79b620cd43728bd9b74496eeeefc P 8d97b360ab544c8ba86d44fc332d7312: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8d97b360ab544c8ba86d44fc332d7312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } } peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } } }
I20260620 18:32:06.376987 12402 tablet_replica.cc:442] T 35ec79b620cd43728bd9b74496eeeefc P 8d97b360ab544c8ba86d44fc332d7312: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:32:06.380617 12398 tablet_replica.cc:439] T 35ec79b620cd43728bd9b74496eeeefc P ab3b9330a0544503a5fd00fa7f77db11: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8d97b360ab544c8ba86d44fc332d7312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d97b360ab544c8ba86d44fc332d7312" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 39489 } } peers { permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 } } peers { permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 } } }
I20260620 18:32:06.380884 12398 tablet_replica.cc:442] T 35ec79b620cd43728bd9b74496eeeefc P ab3b9330a0544503a5fd00fa7f77db11: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:32:06.400637 12435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:06.393881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:48981 (local address 127.10.131.1:39489)
0620 18:32:06.394069 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:06.394075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:06.394742 (+ 667us) server_negotiation.cc:407] Connection header received
0620 18:32:06.395060 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:06.395065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:06.395197 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:06.395306 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:32:06.395950 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:06.397090 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:06.398523 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:06.399186 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:06.399318 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:06.399804 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:32:06.399926 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:32:06.400015 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:06.400128 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 18:32:06.400204 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:32:06.401046 12446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:06.393881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:39489 (local address 127.10.131.1:48981)
0620 18:32:06.394651 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:06.394672 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:06.394855 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:06.395358 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:06.395365 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:06.395408 (+ 43us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:32:06.395800 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:06.395809 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:06.397226 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:06.397229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:06.398379 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:06.398390 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:06.399389 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:06.399394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:06.399573 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:06.399633 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:32:06.400025 (+ 392us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:32:06.400034 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 18:32:06.400099 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 18:32:06.400213 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":1,"thread_start_us":407,"threads_started":1}
I20260620 18:32:06.410399 12322 raft_consensus.cc:1261] T 74f3b50dba41448eb165cfb69bd470c7 P 57e470bede444347bb8661831e02d6b2 [term 1 FOLLOWER]: Refusing update from remote peer 8d97b360ab544c8ba86d44fc332d7312: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:06.410470 12187 raft_consensus.cc:1261] T 74f3b50dba41448eb165cfb69bd470c7 P ab3b9330a0544503a5fd00fa7f77db11 [term 1 FOLLOWER]: Refusing update from remote peer 8d97b360ab544c8ba86d44fc332d7312: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:06.411195 12402 consensus_queue.cc:1035] T 74f3b50dba41448eb165cfb69bd470c7 P 8d97b360ab544c8ba86d44fc332d7312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3b9330a0544503a5fd00fa7f77db11" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:06.411471 12410 consensus_queue.cc:1035] T 74f3b50dba41448eb165cfb69bd470c7 P 8d97b360ab544c8ba86d44fc332d7312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57e470bede444347bb8661831e02d6b2" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:06.465564 11886 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:32:06.465822 11886 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:32:06.468932 11886 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1cb409d78ff4d0d8c15e24ede3ff4bd: Sending DeleteTablet for 3 replicas of tablet 74f3b50dba41448eb165cfb69bd470c7
I20260620 18:32:06.469899 12302 tablet_service.cc:1514] Processing DeleteTablet for tablet 74f3b50dba41448eb165cfb69bd470c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:06 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:58778
I20260620 18:32:06.470151 12163 tablet_service.cc:1514] Processing DeleteTablet for tablet 74f3b50dba41448eb165cfb69bd470c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:06 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:38174
I20260620 18:32:06.470566 12023 tablet_service.cc:1514] Processing DeleteTablet for tablet 74f3b50dba41448eb165cfb69bd470c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:06 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:52798
I20260620 18:32:06.481511 12452 tablet_replica.cc:330] T 74f3b50dba41448eb165cfb69bd470c7 P ab3b9330a0544503a5fd00fa7f77db11: stopping tablet replica
I20260620 18:32:06.482188 12454 tablet_replica.cc:330] T 74f3b50dba41448eb165cfb69bd470c7 P 8d97b360ab544c8ba86d44fc332d7312: stopping tablet replica
I20260620 18:32:06.482326 12451 tablet_replica.cc:330] T 74f3b50dba41448eb165cfb69bd470c7 P 57e470bede444347bb8661831e02d6b2: stopping tablet replica
I20260620 18:32:06.482538 12451 raft_consensus.cc:2229] T 74f3b50dba41448eb165cfb69bd470c7 P 57e470bede444347bb8661831e02d6b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:32:06.482823 12451 raft_consensus.cc:2258] T 74f3b50dba41448eb165cfb69bd470c7 P 57e470bede444347bb8661831e02d6b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:06.482846 12452 raft_consensus.cc:2229] T 74f3b50dba41448eb165cfb69bd470c7 P ab3b9330a0544503a5fd00fa7f77db11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:32:06.482599 12454 raft_consensus.cc:2229] T 74f3b50dba41448eb165cfb69bd470c7 P 8d97b360ab544c8ba86d44fc332d7312 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:32:06.483873 12454 raft_consensus.cc:2258] T 74f3b50dba41448eb165cfb69bd470c7 P 8d97b360ab544c8ba86d44fc332d7312 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:06.483946 12451 ts_tablet_manager.cc:1905] T 74f3b50dba41448eb165cfb69bd470c7 P 57e470bede444347bb8661831e02d6b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:06.484264 12452 raft_consensus.cc:2258] T 74f3b50dba41448eb165cfb69bd470c7 P ab3b9330a0544503a5fd00fa7f77db11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:06.485531 12452 ts_tablet_manager.cc:1905] T 74f3b50dba41448eb165cfb69bd470c7 P ab3b9330a0544503a5fd00fa7f77db11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:06.486296 12454 ts_tablet_manager.cc:1905] T 74f3b50dba41448eb165cfb69bd470c7 P 8d97b360ab544c8ba86d44fc332d7312: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:06.491565 12451 ts_tablet_manager.cc:1918] T 74f3b50dba41448eb165cfb69bd470c7 P 57e470bede444347bb8661831e02d6b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:32:06.491700 12451 log.cc:1196] T 74f3b50dba41448eb165cfb69bd470c7 P 57e470bede444347bb8661831e02d6b2: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-2/wal/wals/74f3b50dba41448eb165cfb69bd470c7
I20260620 18:32:06.492128 12451 ts_tablet_manager.cc:1939] T 74f3b50dba41448eb165cfb69bd470c7 P 57e470bede444347bb8661831e02d6b2: Deleting consensus metadata
I20260620 18:32:06.493661 12452 ts_tablet_manager.cc:1918] T 74f3b50dba41448eb165cfb69bd470c7 P ab3b9330a0544503a5fd00fa7f77db11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:32:06.493824 12452 log.cc:1196] T 74f3b50dba41448eb165cfb69bd470c7 P ab3b9330a0544503a5fd00fa7f77db11: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-1/wal/wals/74f3b50dba41448eb165cfb69bd470c7
I20260620 18:32:06.494366 12452 ts_tablet_manager.cc:1939] T 74f3b50dba41448eb165cfb69bd470c7 P ab3b9330a0544503a5fd00fa7f77db11: Deleting consensus metadata
I20260620 18:32:06.495417 11874 catalog_manager.cc:4841] TS ab3b9330a0544503a5fd00fa7f77db11 (127.10.131.2:40403): tablet 74f3b50dba41448eb165cfb69bd470c7 (table test-table [id=7e7b8ab58b0c4d63a3edc3532cfa4062]) successfully deleted
I20260620 18:32:06.495719 11874 catalog_manager.cc:4841] TS 57e470bede444347bb8661831e02d6b2 (127.10.131.3:37191): tablet 74f3b50dba41448eb165cfb69bd470c7 (table test-table [id=7e7b8ab58b0c4d63a3edc3532cfa4062]) successfully deleted
I20260620 18:32:06.499970 12454 ts_tablet_manager.cc:1918] T 74f3b50dba41448eb165cfb69bd470c7 P 8d97b360ab544c8ba86d44fc332d7312: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:32:06.500206 12454 log.cc:1196] T 74f3b50dba41448eb165cfb69bd470c7 P 8d97b360ab544c8ba86d44fc332d7312: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781980314095189-10764-0/minicluster-data/ts-0/wal/wals/74f3b50dba41448eb165cfb69bd470c7
I20260620 18:32:06.500742 12454 ts_tablet_manager.cc:1939] T 74f3b50dba41448eb165cfb69bd470c7 P 8d97b360ab544c8ba86d44fc332d7312: Deleting consensus metadata
I20260620 18:32:06.501958 11871 catalog_manager.cc:4841] TS 8d97b360ab544c8ba86d44fc332d7312 (127.10.131.1:39489): tablet 74f3b50dba41448eb165cfb69bd470c7 (table test-table [id=7e7b8ab58b0c4d63a3edc3532cfa4062]) successfully deleted
Jun 20 18:32:06 dist-test-slave-cbn3 krb5kdc[11842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980325, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.131.1@KRBTEST.COM
I20260620 18:32:06.515662 12435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:06.492818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52828 (local address 127.10.131.1:39489)
0620 18:32:06.493016 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:06.493024 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:06.495876 (+ 2852us) server_negotiation.cc:407] Connection header received
0620 18:32:06.496199 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:06.496204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:06.496290 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:06.496376 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:06.498294 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:06.499509 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:06.501364 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:06.502075 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:06.502166 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:06.508086 (+ 5920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:06.508101 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:06.508126 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:06.508184 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:06.512428 (+ 4244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:06.513330 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:06.513341 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:06.513346 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:06.513516 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:06.513970 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:06.513975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:06.513978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:06.514445 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:06.514735 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:06.515032 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 18:32:06.515287 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 18:32:06.516546 12089 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52828
I20260620 18:32:06.537585 12427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:06.522059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55966 (local address 127.10.131.62:45203)
0620 18:32:06.522304 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:06.522312 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:06.522841 (+ 529us) server_negotiation.cc:407] Connection header received
0620 18:32:06.523036 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:06.523042 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:06.523121 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:06.523213 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:06.524914 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:06.526071 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:06.527905 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:06.528581 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:06.528702 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:06.531596 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:06.531603 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:06.531608 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:06.531647 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:06.534560 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:06.535503 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:06.535511 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:06.535516 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:06.535709 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:06.536078 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:06.536083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:06.536087 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:06.536373 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:06.536625 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:06.537025 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 18:32:06.537269 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
W20260620 18:32:06.538235 11886 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55966
I20260620 18:32:06.540134 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 11958
I20260620 18:32:06.569069 12435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:06.541313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52844 (local address 127.10.131.1:39489)
0620 18:32:06.541500 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:06.541507 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:06.553438 (+ 11931us) server_negotiation.cc:407] Connection header received
0620 18:32:06.553588 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:06.553593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:06.553690 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:06.553811 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:06.555346 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:06.556489 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:06.558248 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:06.558932 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:06.559067 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:06.562078 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:06.562085 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:06.562091 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:06.562150 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:06.565976 (+ 3826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:06.566801 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:06.566808 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:06.566817 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:06.566992 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:06.567485 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:06.567491 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:06.567494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:06.567885 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:06.568169 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:06.568407 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 18:32:06.568658 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 18:32:06.569708 12089 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52844
W20260620 18:32:06.570266 10764 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 11958: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:32:06.570466 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 11958
I20260620 18:32:06.593611 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 12098
Jun 20 18:32:06 dist-test-slave-cbn3 krb5kdc[11842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980325, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.131.2@KRBTEST.COM
I20260620 18:32:06.609007 12459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:06.594421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38190 (local address 127.10.131.2:40403)
0620 18:32:06.594974 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:06.594979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:06.594995 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:32:06.595050 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:06.595054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:06.595118 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:06.595214 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:06.596475 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:06.597598 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:06.599171 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:06.599581 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:06.599689 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:06.603706 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:06.603721 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:06.603736 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:06.603770 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:06.606628 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:06.607229 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:06.607236 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:06.607238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:06.607368 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:06.607718 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:06.607721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:06.607723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:06.608039 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:06.608236 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:06.608482 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 18:32:06.608670 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":318,"threads_started":1}
W20260620 18:32:06.609658 12228 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38190
W20260620 18:32:06.610314 10764 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 12098: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:32:06.610484 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 12098
I20260620 18:32:06.627992 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 12237
Jun 20 18:32:06 dist-test-slave-cbn3 krb5kdc[11842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980325, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.131.3@KRBTEST.COM
I20260620 18:32:06.643265 12460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:06.628742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58782 (local address 127.10.131.3:37191)
0620 18:32:06.629248 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:06.629253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:06.629269 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:32:06.629321 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:06.629325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:06.629386 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:06.629473 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:06.630635 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:06.631395 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:06.632559 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:06.633093 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:06.633190 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:06.637282 (+ 4092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:06.637297 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:06.637313 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:06.637343 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:06.640284 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:06.640986 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:06.640994 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:06.640996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:06.641149 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:06.641740 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:06.641745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:06.641747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:06.642129 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:06.642488 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:06.642819 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 18:32:06.642977 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":275,"threads_started":1}
W20260620 18:32:06.644025 12367 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58782
W20260620 18:32:06.644692 10764 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 12237: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:32:06.644848 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 12237
I20260620 18:32:06.661536 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 11855
I20260620 18:32:06.662777 11927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:06.775715 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 11855
2026-06-20T18:32:06Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4609 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:32:06 dist-test-slave-cbn3 krb5kdc[12464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:32:06 dist-test-slave-cbn3 krb5kdc[12464](info): set up 2 sockets
Jun 20 18:32:06 dist-test-slave-cbn3 krb5kdc[12464](info): commencing operation
krb5kdc: starting...
W20260620 18:32:07.853931 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.002s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:32:07 dist-test-slave-cbn3 krb5kdc[12464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:32:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:32:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.131.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
I20260620 18:32:08.040262 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:43781
--webserver_interface=127.10.131.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:33085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:43781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:08.366875 12477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:08.367282 12477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:08.367473 12477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:08.375000 12477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:32:08.375200 12477 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 18:32:08.375288 12477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:08.375388 12477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:08.375502 12477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:32:08.375599 12477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:32:08.385836 12477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:33085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:43781
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:43781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:08.387264 12477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:08.388649 12477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:08.401088 12482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:08.401163 12483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:08.401131 12485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:08.401824 12477 server_base.cc:1034] running on GCE node
I20260620 18:32:08.402930 12477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:08.404441 12477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:08.405718 12477 hybrid_clock.cc:648] HybridClock initialized: now 1781980328405653 us; error 40 us; skew 500 ppm
I20260620 18:32:08.406111 12477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:08 dist-test-slave-cbn3 krb5kdc[12464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980328, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:08.409488 12477 init.cc:377] Logged in from keytab as kudu/127.10.131.62@KRBTEST.COM (short username kudu)
I20260620 18:32:08.411069 12477 webserver.cc:460] Webserver started at http://127.10.131.62:45019/ using document root <none> and password file <none>
I20260620 18:32:08.411865 12477 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:08.412009 12477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:08.412349 12477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:08.415616 12477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "f77c29104bae4b95b9f3c560d0589bb9"
format_stamp: "Formatted at 2026-06-20 18:32:08 on dist-test-slave-cbn3"
server_key: "b7814f326550de17c1189579a3ecd70f"
server_key_iv: "7728f6832818b2611138af6b37089cee"
server_key_version: "encryptionkey@0"
I20260620 18:32:08.416463 12477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "f77c29104bae4b95b9f3c560d0589bb9"
format_stamp: "Formatted at 2026-06-20 18:32:08 on dist-test-slave-cbn3"
server_key: "b7814f326550de17c1189579a3ecd70f"
server_key_iv: "7728f6832818b2611138af6b37089cee"
server_key_version: "encryptionkey@0"
I20260620 18:32:08.422363 12477 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 18:32:08.426168 12492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:08.427670 12477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:32:08.427882 12477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "f77c29104bae4b95b9f3c560d0589bb9"
format_stamp: "Formatted at 2026-06-20 18:32:08 on dist-test-slave-cbn3"
server_key: "b7814f326550de17c1189579a3ecd70f"
server_key_iv: "7728f6832818b2611138af6b37089cee"
server_key_version: "encryptionkey@0"
I20260620 18:32:08.428117 12477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:08.454057 12477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:08.458578 12477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:08.458871 12477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:08.480721 12477 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:43781
I20260620 18:32:08.480764 12553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:43781 every 8 connection(s)
I20260620 18:32:08.482090 12477 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:32:08.485940 12554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:08.488374 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 12477
I20260620 18:32:08.488605 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:32:08.489513 10764 external_mini_cluster.cc:1428] Setting key 9dab65184f7af43deb32bf5389c6fd25
I20260620 18:32:08.499740 12554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f77c29104bae4b95b9f3c560d0589bb9: Bootstrap starting.
I20260620 18:32:08.504022 12554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f77c29104bae4b95b9f3c560d0589bb9: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:08.505411 12554 log.cc:824] T 00000000000000000000000000000000 P f77c29104bae4b95b9f3c560d0589bb9: Log is configured to *not* fsync() on all Append() calls
Jun 20 18:32:08 dist-test-slave-cbn3 krb5kdc[12464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:32:08.508491 12554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f77c29104bae4b95b9f3c560d0589bb9: No bootstrap required, opened a new log
I20260620 18:32:08.514024 12557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:08.491770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52478 (local address 127.10.131.62:43781)
0620 18:32:08.492678 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:08.492691 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:08.492732 (+ 41us) server_negotiation.cc:407] Connection header received
0620 18:32:08.494444 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:08.494466 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:08.494850 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:08.495162 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:08.497222 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:08.498599 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:08.500503 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:08.501288 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:08.501461 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:08.506568 (+ 5107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:08.506577 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:08.506592 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:08.506640 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:08.509844 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:08.510567 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:08.510580 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:08.510586 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:08.510748 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:08.511148 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:08.511156 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:08.511161 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:08.511680 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:08.512042 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:08.512474 (+ 432us) server_negotiation.cc:299] Negotiation successful
0620 18:32:08.513000 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":425,"threads_started":1}
I20260620 18:32:08.516791 12554 raft_consensus.cc:348] T 00000000000000000000000000000000 P f77c29104bae4b95b9f3c560d0589bb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f77c29104bae4b95b9f3c560d0589bb9" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 43781 } }
I20260620 18:32:08.517324 12554 raft_consensus.cc:374] T 00000000000000000000000000000000 P f77c29104bae4b95b9f3c560d0589bb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:08.517439 12554 raft_consensus.cc:729] T 00000000000000000000000000000000 P f77c29104bae4b95b9f3c560d0589bb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f77c29104bae4b95b9f3c560d0589bb9, State: Initialized, Role: FOLLOWER
I20260620 18:32:08.518050 12554 consensus_queue.cc:260] T 00000000000000000000000000000000 P f77c29104bae4b95b9f3c560d0589bb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f77c29104bae4b95b9f3c560d0589bb9" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 43781 } }
I20260620 18:32:08.518385 12554 raft_consensus.cc:388] T 00000000000000000000000000000000 P f77c29104bae4b95b9f3c560d0589bb9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:32:08.518527 12554 raft_consensus.cc:482] T 00000000000000000000000000000000 P f77c29104bae4b95b9f3c560d0589bb9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:32:08.518715 12554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f77c29104bae4b95b9f3c560d0589bb9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:08.521406 12554 raft_consensus.cc:504] T 00000000000000000000000000000000 P f77c29104bae4b95b9f3c560d0589bb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f77c29104bae4b95b9f3c560d0589bb9" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 43781 } }
I20260620 18:32:08.522024 12554 leader_election.cc:302] T 00000000000000000000000000000000 P f77c29104bae4b95b9f3c560d0589bb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f77c29104bae4b95b9f3c560d0589bb9; no voters:
I20260620 18:32:08.522665 12554 leader_election.cc:288] T 00000000000000000000000000000000 P f77c29104bae4b95b9f3c560d0589bb9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:32:08.522773 12559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f77c29104bae4b95b9f3c560d0589bb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:08.523622 12559 raft_consensus.cc:686] T 00000000000000000000000000000000 P f77c29104bae4b95b9f3c560d0589bb9 [term 1 LEADER]: Becoming Leader. State: Replica: f77c29104bae4b95b9f3c560d0589bb9, State: Running, Role: LEADER
I20260620 18:32:08.524214 12559 consensus_queue.cc:237] T 00000000000000000000000000000000 P f77c29104bae4b95b9f3c560d0589bb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f77c29104bae4b95b9f3c560d0589bb9" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 43781 } }
I20260620 18:32:08.525115 12554 sys_catalog.cc:564] T 00000000000000000000000000000000 P f77c29104bae4b95b9f3c560d0589bb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:32:08.527328 12561 sys_catalog.cc:455] T 00000000000000000000000000000000 P f77c29104bae4b95b9f3c560d0589bb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f77c29104bae4b95b9f3c560d0589bb9. Latest consensus state: current_term: 1 leader_uuid: "f77c29104bae4b95b9f3c560d0589bb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f77c29104bae4b95b9f3c560d0589bb9" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 43781 } } }
I20260620 18:32:08.527824 12561 sys_catalog.cc:458] T 00000000000000000000000000000000 P f77c29104bae4b95b9f3c560d0589bb9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:08.528342 12560 sys_catalog.cc:455] T 00000000000000000000000000000000 P f77c29104bae4b95b9f3c560d0589bb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f77c29104bae4b95b9f3c560d0589bb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f77c29104bae4b95b9f3c560d0589bb9" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 43781 } } }
I20260620 18:32:08.528765 12560 sys_catalog.cc:458] T 00000000000000000000000000000000 P f77c29104bae4b95b9f3c560d0589bb9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:08.529670 12568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:32:08.534687 12568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:32:08.543679 12568 catalog_manager.cc:1347] Generated new cluster ID: 60e3af7029ad4a21afe1274422ff3730
I20260620 18:32:08.543808 12568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:32:08.552291 12568 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:32:08.554059 12568 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:32:08.562094 12568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f77c29104bae4b95b9f3c560d0589bb9: Generated new TSK 0
I20260620 18:32:08.563073 12568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.131.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab.
Entry for principal kudu/127.10.131.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab.
I20260620 18:32:08.612322 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.1:0
--local_ip_for_outbound_sockets=127.10.131.1
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:43781
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:33085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:08.946959 12580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:08.947268 12580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:08.947386 12580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:08.954998 12580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:08.955124 12580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:08.955215 12580 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 18:32:08.955271 12580 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:32:08.955332 12580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.1
I20260620 18:32:08.965396 12580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:33085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.131.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.131.62:43781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.1
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:08.966668 12580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:08.967983 12580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:08.981930 12588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:08.982110 12585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:08.982049 12586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:08.982961 12580 server_base.cc:1034] running on GCE node
I20260620 18:32:08.983659 12580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:08.984615 12580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:08.985833 12580 hybrid_clock.cc:648] HybridClock initialized: now 1781980328985756 us; error 73 us; skew 500 ppm
I20260620 18:32:08.986158 12580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:08 dist-test-slave-cbn3 krb5kdc[12464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980328, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:08.989277 12580 init.cc:377] Logged in from keytab as kudu/127.10.131.1@KRBTEST.COM (short username kudu)
I20260620 18:32:08.990832 12580 webserver.cc:460] Webserver started at http://127.10.131.1:38729/ using document root <none> and password file <none>
I20260620 18:32:08.991497 12580 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:08.991595 12580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:08.991848 12580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:08.994802 12580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-0/data/instance:
uuid: "38da097b32ae4db9a2d2ad6f5e03d266"
format_stamp: "Formatted at 2026-06-20 18:32:08 on dist-test-slave-cbn3"
server_key: "c422932f716a8ff5714ab64b7404448c"
server_key_iv: "b67db21756ef44b5644b4a0b0634679d"
server_key_version: "encryptionkey@0"
I20260620 18:32:08.995524 12580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance:
uuid: "38da097b32ae4db9a2d2ad6f5e03d266"
format_stamp: "Formatted at 2026-06-20 18:32:08 on dist-test-slave-cbn3"
server_key: "c422932f716a8ff5714ab64b7404448c"
server_key_iv: "b67db21756ef44b5644b4a0b0634679d"
server_key_version: "encryptionkey@0"
I20260620 18:32:09.000941 12580 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 18:32:09.004426 12595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:09.006008 12580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 18:32:09.006155 12580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-0/wal
uuid: "38da097b32ae4db9a2d2ad6f5e03d266"
format_stamp: "Formatted at 2026-06-20 18:32:08 on dist-test-slave-cbn3"
server_key: "c422932f716a8ff5714ab64b7404448c"
server_key_iv: "b67db21756ef44b5644b4a0b0634679d"
server_key_version: "encryptionkey@0"
I20260620 18:32:09.006332 12580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:09.021984 12580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:09.026080 12580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:09.026376 12580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:09.029428 12580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:09.029536 12580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:09.029668 12580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:09.029772 12580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:32:09 dist-test-slave-cbn3 krb5kdc[12464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980328, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:32:09.055670 12605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:09.030819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:43781 (local address 127.10.131.1:49373)
0620 18:32:09.031978 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:09.032038 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:09.035215 (+ 3177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:09.036259 (+ 1044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:09.036276 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:09.037141 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:09.038004 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:09.038027 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:09.039813 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:09.039823 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:09.041442 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:09.041462 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:09.042672 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:09.042682 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:09.042996 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:09.043732 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:09.043754 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:09.046986 (+ 3232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:09.051298 (+ 4312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:09.051308 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:09.051326 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:09.051774 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:09.052174 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:09.052180 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:09.052184 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:09.052391 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:09.053024 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:09.053033 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:09.053615 (+ 582us) client_negotiation.cc:769] Sending connection context
0620 18:32:09.053966 (+ 351us) client_negotiation.cc:240] Negotiation successful
0620 18:32:09.054554 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"thread_start_us":425,"threads_started":1}
I20260620 18:32:09.056740 12604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:09.030031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:49373 (local address 127.10.131.62:43781)
0620 18:32:09.032929 (+ 2898us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:09.032936 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:09.032961 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:32:09.035664 (+ 2703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:09.035673 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:09.035781 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:09.035910 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:09.038313 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:09.039608 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:09.041647 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:09.042387 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:09.042561 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:09.047299 (+ 4738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:09.047307 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:09.047314 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:09.047354 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:09.051137 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:09.051915 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:09.051925 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:09.051927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:09.052050 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:09.052520 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:09.052528 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:09.052535 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:09.052823 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:09.053039 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:09.056274 (+ 3235us) server_negotiation.cc:299] Negotiation successful
0620 18:32:09.056489 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2733,"thread_start_us":2643,"threads_started":1}
I20260620 18:32:09.080018 12604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:09.064026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:60313 (local address 127.10.131.62:43781)
0620 18:32:09.064230 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:09.064236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:09.065914 (+ 1678us) server_negotiation.cc:407] Connection header received
0620 18:32:09.066201 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:09.066208 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:09.066306 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:09.066455 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:09.067912 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:09.068988 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:09.070344 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:09.070764 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:09.070901 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:09.072830 (+ 1929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:09.072887 (+ 57us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:09.072895 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:09.072939 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:09.076321 (+ 3382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:09.077419 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:09.077424 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:09.077426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:09.077547 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:09.078063 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:09.078071 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:09.078075 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:09.078495 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:09.078667 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:09.079477 (+ 810us) server_negotiation.cc:299] Negotiation successful
0620 18:32:09.079743 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:32:09.080313 12677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:09.063543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:43781 (local address 127.10.131.1:60313)
0620 18:32:09.065798 (+ 2255us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:09.065819 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:09.066035 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:09.066506 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:09.066512 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:09.067162 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:09.067732 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:09.067751 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:09.069153 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:09.069158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:09.070183 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:09.070194 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:09.071004 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:09.071011 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:09.071174 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:09.071844 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:09.071862 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:09.072625 (+ 763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:09.076500 (+ 3875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:09.076510 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:09.076516 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:09.077234 (+ 718us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:09.077716 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:09.077723 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:09.077728 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:09.077921 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:09.078627 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:09.078632 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:09.078884 (+ 252us) client_negotiation.cc:769] Sending connection context
0620 18:32:09.079088 (+ 204us) client_negotiation.cc:240] Negotiation successful
0620 18:32:09.079632 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2067,"thread_start_us":1794,"threads_started":1}
I20260620 18:32:09.083034 12580 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.1:41907
I20260620 18:32:09.083057 12715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.1:41907 every 8 connection(s)
I20260620 18:32:09.084396 12580 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
I20260620 18:32:09.086506 12716 heartbeater.cc:347] Connected to a master server at 127.10.131.62:43781
I20260620 18:32:09.086689 12716 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:09.087265 12716 heartbeater.cc:511] Master 127.10.131.62:43781 requested a full tablet report, sending...
I20260620 18:32:09.089337 12509 ts_manager.cc:194] Registered new tserver with Master: 38da097b32ae4db9a2d2ad6f5e03d266 (127.10.131.1:41907)
I20260620 18:32:09.090795 12509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.1@KRBTEST.COM'} at 127.10.131.1:49373
I20260620 18:32:09.091807 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 12580
I20260620 18:32:09.092010 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance
I20260620 18:32:09.092687 10764 external_mini_cluster.cc:1428] Setting key ee08b9055b40a5df5b609c615e2e6ea6
WARNING: no policy specified for kudu/127.10.131.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab.
Entry for principal kudu/127.10.131.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab.
I20260620 18:32:09.131729 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.2:0
--local_ip_for_outbound_sockets=127.10.131.2
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:43781
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:33085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:09.497377 12720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:09.497692 12720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:09.497871 12720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:09.505690 12720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:09.505847 12720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:09.506024 12720 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 18:32:09.506120 12720 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:32:09.506263 12720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.2
I20260620 18:32:09.516388 12720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:33085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.131.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.131.62:43781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.2
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:09.517705 12720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:09.519032 12720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:09.533171 12726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:09.533171 12728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:09.533833 12725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:09.534967 12720 server_base.cc:1034] running on GCE node
I20260620 18:32:09.535557 12720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:09.536471 12720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:09.537714 12720 hybrid_clock.cc:648] HybridClock initialized: now 1781980329537674 us; error 34 us; skew 500 ppm
I20260620 18:32:09.538038 12720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:09 dist-test-slave-cbn3 krb5kdc[12464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980329, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:09.541567 12720 init.cc:377] Logged in from keytab as kudu/127.10.131.2@KRBTEST.COM (short username kudu)
I20260620 18:32:09.543025 12720 webserver.cc:460] Webserver started at http://127.10.131.2:43433/ using document root <none> and password file <none>
I20260620 18:32:09.543706 12720 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:09.543855 12720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:09.544184 12720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:09.547433 12720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-1/data/instance:
uuid: "eeee1578317046a7acff34fd57ab3022"
format_stamp: "Formatted at 2026-06-20 18:32:09 on dist-test-slave-cbn3"
server_key: "c0997874827402cb416f5e06fb545cef"
server_key_iv: "a1c20c71d2a75d76f008ceaa8f5b0eeb"
server_key_version: "encryptionkey@0"
I20260620 18:32:09.548213 12720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance:
uuid: "eeee1578317046a7acff34fd57ab3022"
format_stamp: "Formatted at 2026-06-20 18:32:09 on dist-test-slave-cbn3"
server_key: "c0997874827402cb416f5e06fb545cef"
server_key_iv: "a1c20c71d2a75d76f008ceaa8f5b0eeb"
server_key_version: "encryptionkey@0"
I20260620 18:32:09.553417 12720 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260620 18:32:09.556813 12735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:09.558396 12720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:32:09.558563 12720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-1/wal
uuid: "eeee1578317046a7acff34fd57ab3022"
format_stamp: "Formatted at 2026-06-20 18:32:09 on dist-test-slave-cbn3"
server_key: "c0997874827402cb416f5e06fb545cef"
server_key_iv: "a1c20c71d2a75d76f008ceaa8f5b0eeb"
server_key_version: "encryptionkey@0"
I20260620 18:32:09.558774 12720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:09.578632 12720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:09.583060 12720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:09.583359 12720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:09.586162 12720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:09.586300 12720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:09.586427 12720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:09.586520 12720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:32:09 dist-test-slave-cbn3 krb5kdc[12464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980329, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:32:09.613332 12745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:09.587599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:43781 (local address 127.10.131.2:48573)
0620 18:32:09.588634 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:09.588688 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:09.591730 (+ 3042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:09.592722 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:09.592737 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:09.593614 (+ 877us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:09.594441 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:09.594462 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:09.595902 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:09.595906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:09.598046 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:09.598060 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:09.599398 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:09.599405 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:09.599598 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:09.600403 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:09.600448 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:09.603717 (+ 3269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:09.607812 (+ 4095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:09.607822 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:09.607844 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:09.608552 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:09.609061 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:09.609066 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:09.609070 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:09.609413 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:09.610216 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:09.610227 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:09.610771 (+ 544us) client_negotiation.cc:769] Sending connection context
0620 18:32:09.611098 (+ 327us) client_negotiation.cc:240] Negotiation successful
0620 18:32:09.611700 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":451,"threads_started":1}
I20260620 18:32:09.614502 12744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:09.586778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:48573 (local address 127.10.131.62:43781)
0620 18:32:09.589596 (+ 2818us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:09.589600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:09.589620 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:32:09.592122 (+ 2502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:09.592130 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:09.592230 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:09.592381 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:09.594630 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:09.595725 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:09.598221 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:09.599091 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:09.599241 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:09.604031 (+ 4790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:09.604039 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:09.604046 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:09.604095 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:09.607604 (+ 3509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:09.608717 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:09.608726 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:09.608730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:09.608911 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:09.609564 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:09.609571 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:09.609576 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:09.609972 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:09.610137 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:09.613857 (+ 3720us) server_negotiation.cc:299] Negotiation successful
0620 18:32:09.614114 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2702,"thread_start_us":2613,"threads_started":1}
I20260620 18:32:09.640554 12744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:09.622306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:39303 (local address 127.10.131.62:43781)
0620 18:32:09.622489 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:09.622497 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:09.623200 (+ 703us) server_negotiation.cc:407] Connection header received
0620 18:32:09.623383 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:09.623386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:09.623449 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:09.623529 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:09.625110 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:09.625969 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:09.628434 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:09.628977 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:09.629145 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:09.632353 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:09.632359 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:09.632365 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:09.632410 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:09.637164 (+ 4754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:09.638273 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:09.638278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:09.638282 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:09.638435 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:09.638951 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:09.638957 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:09.638961 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:09.639433 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:09.639579 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:09.639983 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 18:32:09.640215 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 18:32:09.640682 12817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:09.622306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:43781 (local address 127.10.131.2:39303)
0620 18:32:09.623068 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:09.623099 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:09.623263 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:09.623605 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:09.623612 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:09.624371 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:09.624908 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:09.624929 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:09.626111 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:09.626118 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:09.628281 (+ 2163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:09.628300 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:09.629192 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:09.629221 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:09.629456 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:09.630350 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:09.630383 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:09.632088 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:09.637367 (+ 5279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:09.637375 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:09.637381 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:09.638117 (+ 736us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:09.638562 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:09.638568 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:09.638571 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:09.638777 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:09.639537 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:09.639542 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:09.639732 (+ 190us) client_negotiation.cc:769] Sending connection context
0620 18:32:09.639893 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 18:32:09.640228 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":359,"threads_started":1}
I20260620 18:32:09.649322 12720 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.2:36507
I20260620 18:32:09.649410 12855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.2:36507 every 8 connection(s)
I20260620 18:32:09.650501 12720 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
I20260620 18:32:09.652545 12856 heartbeater.cc:347] Connected to a master server at 127.10.131.62:43781
I20260620 18:32:09.652742 12856 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:09.653379 12856 heartbeater.cc:511] Master 127.10.131.62:43781 requested a full tablet report, sending...
I20260620 18:32:09.653566 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 12720
I20260620 18:32:09.653738 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance
I20260620 18:32:09.654241 10764 external_mini_cluster.cc:1428] Setting key eab3525ea85e28e16b45742cd17e76c5
I20260620 18:32:09.654722 12509 ts_manager.cc:194] Registered new tserver with Master: eeee1578317046a7acff34fd57ab3022 (127.10.131.2:36507)
I20260620 18:32:09.656085 12509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.2@KRBTEST.COM'} at 127.10.131.2:48573
WARNING: no policy specified for kudu/127.10.131.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab.
Entry for principal kudu/127.10.131.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab.
I20260620 18:32:09.695988 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.3:0
--local_ip_for_outbound_sockets=127.10.131.3
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:43781
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:33085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:10.034849 12860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:10.035171 12860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:10.035382 12860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:10.043612 12860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:10.043730 12860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:10.043907 12860 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 18:32:10.043990 12860 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:32:10.044129 12860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.3
I20260620 18:32:10.054287 12860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:33085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.131.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.131.62:43781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.3
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:10.055568 12860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:10.056880 12860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:10.069913 12865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:10.070003 12868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:10.070076 12866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:10.071616 12860 server_base.cc:1034] running on GCE node
I20260620 18:32:10.072312 12860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:10.073395 12860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:10.074637 12860 hybrid_clock.cc:648] HybridClock initialized: now 1781980330074584 us; error 47 us; skew 500 ppm
I20260620 18:32:10.074961 12860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:10 dist-test-slave-cbn3 krb5kdc[12464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980330, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:10.082955 12860 init.cc:377] Logged in from keytab as kudu/127.10.131.3@KRBTEST.COM (short username kudu)
I20260620 18:32:10.086817 12860 webserver.cc:460] Webserver started at http://127.10.131.3:41177/ using document root <none> and password file <none>
I20260620 18:32:10.087587 12860 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:10.087734 12860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:10.088078 12860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:10.091151 12860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-2/data/instance:
uuid: "5b244d023c7247219f0d39a360b3e3e0"
format_stamp: "Formatted at 2026-06-20 18:32:10 on dist-test-slave-cbn3"
server_key: "ba813bb993e4f9537cd6e928ef109a5d"
server_key_iv: "575607b2b384f71165e27d30e0b619d5"
server_key_version: "encryptionkey@0"
I20260620 18:32:10.091955 12860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance:
uuid: "5b244d023c7247219f0d39a360b3e3e0"
format_stamp: "Formatted at 2026-06-20 18:32:10 on dist-test-slave-cbn3"
server_key: "ba813bb993e4f9537cd6e928ef109a5d"
server_key_iv: "575607b2b384f71165e27d30e0b619d5"
server_key_version: "encryptionkey@0"
I20260620 18:32:10.094117 12716 heartbeater.cc:503] Master 127.10.131.62:43781 was elected leader, sending a full tablet report...
I20260620 18:32:10.097923 12860 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260620 18:32:10.101428 12875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:10.102767 12860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:32:10.102941 12860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-2/wal
uuid: "5b244d023c7247219f0d39a360b3e3e0"
format_stamp: "Formatted at 2026-06-20 18:32:10 on dist-test-slave-cbn3"
server_key: "ba813bb993e4f9537cd6e928ef109a5d"
server_key_iv: "575607b2b384f71165e27d30e0b619d5"
server_key_version: "encryptionkey@0"
I20260620 18:32:10.103121 12860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:10.139119 12860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:10.143985 12860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:10.144323 12860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:10.147390 12860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:10.147547 12860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:10.147681 12860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:10.147778 12860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:32:10 dist-test-slave-cbn3 krb5kdc[12464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980330, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:32:10.173158 12884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:10.148168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:44989 (local address 127.10.131.62:43781)
0620 18:32:10.148761 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:10.148769 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:10.150118 (+ 1349us) server_negotiation.cc:407] Connection header received
0620 18:32:10.152927 (+ 2809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:10.152935 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:10.153047 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:10.153191 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:10.155648 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.156789 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.159719 (+ 2930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.160155 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.160310 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:10.165309 (+ 4999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:10.165316 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:10.165322 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:10.165362 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:10.168944 (+ 3582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:10.170051 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:10.170061 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:10.170068 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:10.170237 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:10.170873 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:10.170879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:10.170883 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:10.171296 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:10.171451 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:10.172463 (+ 1012us) server_negotiation.cc:299] Negotiation successful
0620 18:32:10.172686 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":344,"threads_started":1}
I20260620 18:32:10.174188 12885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:10.148965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:43781 (local address 127.10.131.3:44989)
0620 18:32:10.149906 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:10.149963 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:10.152546 (+ 2583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:10.153462 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:10.153470 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:10.154553 (+ 1083us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:10.155454 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:10.155477 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.157107 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.157112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:10.159547 (+ 2435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:10.159561 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.160421 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.160429 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:10.160668 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:10.161599 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:10.161647 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:10.165019 (+ 3372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:10.169114 (+ 4095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:10.169127 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:10.169157 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:10.169868 (+ 711us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:10.170386 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:10.170395 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:10.170400 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:10.170731 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:10.171445 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:10.171454 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:10.171981 (+ 527us) client_negotiation.cc:769] Sending connection context
0620 18:32:10.172380 (+ 399us) client_negotiation.cc:240] Negotiation successful
0620 18:32:10.173005 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":436,"threads_started":1}
I20260620 18:32:10.195290 12884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:10.180125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:45733 (local address 127.10.131.62:43781)
0620 18:32:10.180335 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:10.180343 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:10.181336 (+ 993us) server_negotiation.cc:407] Connection header received
0620 18:32:10.181553 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:10.181558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:10.181630 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:10.181775 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:10.182759 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.183484 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.185018 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.185666 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.185827 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:10.188581 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:10.188591 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:10.188598 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:10.188650 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:10.192193 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:10.193081 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:10.193089 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:10.193096 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:10.193269 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:10.193663 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:10.193671 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:10.193677 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:10.194034 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:10.194193 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:10.194634 (+ 441us) server_negotiation.cc:299] Negotiation successful
0620 18:32:10.194869 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:32:10.195412 12945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:10.180207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:43781 (local address 127.10.131.3:45733)
0620 18:32:10.181153 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:10.181187 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:10.181377 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:10.181810 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:10.181815 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:10.182300 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:10.182591 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:10.182600 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.183633 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.183639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:10.184777 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:10.184789 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.185887 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.185893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:10.186138 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:10.187028 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:10.187056 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:10.188329 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:10.192361 (+ 4032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:10.192366 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:10.192370 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:10.192950 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:10.193395 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:10.193400 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:10.193404 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:10.193535 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:10.194170 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:10.194173 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:10.194325 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 18:32:10.194455 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 18:32:10.194798 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"thread_start_us":478,"threads_started":1}
I20260620 18:32:10.205080 12860 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.3:35431
I20260620 18:32:10.205170 12995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.3:35431 every 8 connection(s)
I20260620 18:32:10.206368 12860 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
I20260620 18:32:10.208361 12996 heartbeater.cc:347] Connected to a master server at 127.10.131.62:43781
I20260620 18:32:10.208595 12996 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:10.209316 12996 heartbeater.cc:511] Master 127.10.131.62:43781 requested a full tablet report, sending...
I20260620 18:32:10.210712 12509 ts_manager.cc:194] Registered new tserver with Master: 5b244d023c7247219f0d39a360b3e3e0 (127.10.131.3:35431)
I20260620 18:32:10.211561 12509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.3@KRBTEST.COM'} at 127.10.131.3:44989
I20260620 18:32:10.215379 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 12860
I20260620 18:32:10.215536 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance
I20260620 18:32:10.215997 10764 external_mini_cluster.cc:1428] Setting key 90ab1193b9ced37956fcc302c53ab077
I20260620 18:32:10.219061 10764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 18:32:10 dist-test-slave-cbn3 krb5kdc[12464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 18:32:10 dist-test-slave-cbn3 krb5kdc[12464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:32:10.266542 12884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:10.252570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52494 (local address 127.10.131.62:43781)
0620 18:32:10.252836 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:10.252864 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:10.253188 (+ 324us) server_negotiation.cc:407] Connection header received
0620 18:32:10.253430 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:10.253435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:10.253498 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:10.253615 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:10.255034 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.255766 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.257047 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.257449 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.257597 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:10.261391 (+ 3794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:10.261396 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:10.261400 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:10.261433 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:10.264014 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:10.264650 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:10.264654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:10.264657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:10.264771 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:10.265178 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:10.265188 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:10.265190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:10.265445 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:10.265711 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:10.266004 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 18:32:10.266208 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 18:32:10.270599 12509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:32:10.277818 12509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:32:10.302932 13012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:10.291597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59370 (local address 127.10.131.3:35431)
0620 18:32:10.292157 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:10.292163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:10.293543 (+ 1380us) server_negotiation.cc:407] Connection header received
0620 18:32:10.293840 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:10.293848 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:10.294091 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:10.294262 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:10.295303 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.296831 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.299605 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.301181 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.301289 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:10.301533 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:10.302291 (+ 758us) server_negotiation.cc:299] Negotiation successful
0620 18:32:10.302483 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":432,"threads_started":1}
I20260620 18:32:10.303916 13010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:10.290695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:35431 (local address 127.0.0.1:59370)
0620 18:32:10.293414 (+ 2719us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:10.293435 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:10.293650 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:10.294604 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:10.294609 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:10.294630 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:10.295073 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:10.295087 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.297027 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.297032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:10.299420 (+ 2388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:10.299435 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.301714 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.301721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:10.301969 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:10.302040 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 18:32:10.303267 (+ 1227us) client_negotiation.cc:240] Negotiation successful
0620 18:32:10.303519 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2558,"thread_start_us":2349,"threads_started":1}
I20260620 18:32:10.307140 13009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:10.290508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:41907 (local address 127.0.0.1:60390)
0620 18:32:10.291244 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:10.291286 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:10.291465 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:10.292924 (+ 1459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:10.292939 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:10.292987 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:10.293531 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:10.293545 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.296666 (+ 3121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.296671 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:10.300759 (+ 4088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:10.300783 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.306242 (+ 5459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.306249 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:10.306437 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:10.306476 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 18:32:10.306570 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 18:32:10.306732 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":635,"threads_started":1}
I20260620 18:32:10.310506 13011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:10.291057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60390 (local address 127.10.131.1:41907)
0620 18:32:10.292109 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:10.292118 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:10.292145 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:32:10.292265 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:10.292276 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:10.292576 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:10.292877 (+ 301us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:10.294857 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.296487 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.301010 (+ 4523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.306007 (+ 4997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.306176 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:10.306386 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:10.309635 (+ 3249us) server_negotiation.cc:299] Negotiation successful
0620 18:32:10.310054 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":913,"thread_start_us":750,"threads_started":1}
I20260620 18:32:10.312383 12926 tablet_service.cc:1467] Processing CreateTablet for tablet 8764ebe797314512a7556b2bcd4af407 (DEFAULT_TABLE table=test-table [id=bf9e1bda58a54ff59399eb4d84f05b54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:10.313949 12926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8764ebe797314512a7556b2bcd4af407. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:10.313977 12646 tablet_service.cc:1467] Processing CreateTablet for tablet 8764ebe797314512a7556b2bcd4af407 (DEFAULT_TABLE table=test-table [id=bf9e1bda58a54ff59399eb4d84f05b54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:10.316095 12646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8764ebe797314512a7556b2bcd4af407. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:10.316771 13010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:10.293472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:36507 (local address 127.0.0.1:50920)
0620 18:32:10.304258 (+ 10786us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:10.304278 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:10.304370 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:10.310199 (+ 5829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:10.310204 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:10.310233 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:10.310407 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:10.310419 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.312165 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.312172 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:10.314034 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:10.314063 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.316005 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.316012 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:10.316213 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:10.316247 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 18:32:10.316339 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 18:32:10.316490 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10681,"thread_start_us":14510,"threads_started":1}
I20260620 18:32:10.317535 13014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:10.293691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50920 (local address 127.10.131.2:36507)
0620 18:32:10.309327 (+ 15636us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:10.309337 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:10.309366 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:32:10.309470 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:10.309478 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:10.309758 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:10.309946 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:10.310604 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.312000 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.314240 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.315559 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.315703 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:10.315900 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:10.317069 (+ 1169us) server_negotiation.cc:299] Negotiation successful
0620 18:32:10.317258 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15469,"thread_start_us":15332,"threads_started":1}
I20260620 18:32:10.324599 12786 tablet_service.cc:1467] Processing CreateTablet for tablet 8764ebe797314512a7556b2bcd4af407 (DEFAULT_TABLE table=test-table [id=bf9e1bda58a54ff59399eb4d84f05b54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:10.326048 12786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8764ebe797314512a7556b2bcd4af407. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:10.326619 13015 tablet_bootstrap.cc:492] T 8764ebe797314512a7556b2bcd4af407 P 5b244d023c7247219f0d39a360b3e3e0: Bootstrap starting.
I20260620 18:32:10.329545 13016 tablet_bootstrap.cc:492] T 8764ebe797314512a7556b2bcd4af407 P 38da097b32ae4db9a2d2ad6f5e03d266: Bootstrap starting.
I20260620 18:32:10.330430 13015 tablet_bootstrap.cc:654] T 8764ebe797314512a7556b2bcd4af407 P 5b244d023c7247219f0d39a360b3e3e0: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:10.331725 13015 log.cc:824] T 8764ebe797314512a7556b2bcd4af407 P 5b244d023c7247219f0d39a360b3e3e0: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:10.333467 13016 tablet_bootstrap.cc:654] T 8764ebe797314512a7556b2bcd4af407 P 38da097b32ae4db9a2d2ad6f5e03d266: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:10.334978 13016 log.cc:824] T 8764ebe797314512a7556b2bcd4af407 P 38da097b32ae4db9a2d2ad6f5e03d266: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:10.338157 13016 tablet_bootstrap.cc:492] T 8764ebe797314512a7556b2bcd4af407 P 38da097b32ae4db9a2d2ad6f5e03d266: No bootstrap required, opened a new log
I20260620 18:32:10.338424 13016 ts_tablet_manager.cc:1397] T 8764ebe797314512a7556b2bcd4af407 P 38da097b32ae4db9a2d2ad6f5e03d266: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 18:32:10.339286 13015 tablet_bootstrap.cc:492] T 8764ebe797314512a7556b2bcd4af407 P 5b244d023c7247219f0d39a360b3e3e0: No bootstrap required, opened a new log
I20260620 18:32:10.339511 13019 tablet_bootstrap.cc:492] T 8764ebe797314512a7556b2bcd4af407 P eeee1578317046a7acff34fd57ab3022: Bootstrap starting.
I20260620 18:32:10.339536 13015 ts_tablet_manager.cc:1397] T 8764ebe797314512a7556b2bcd4af407 P 5b244d023c7247219f0d39a360b3e3e0: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260620 18:32:10.342325 13019 tablet_bootstrap.cc:654] T 8764ebe797314512a7556b2bcd4af407 P eeee1578317046a7acff34fd57ab3022: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:10.343986 13019 log.cc:824] T 8764ebe797314512a7556b2bcd4af407 P eeee1578317046a7acff34fd57ab3022: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:10.346391 13019 tablet_bootstrap.cc:492] T 8764ebe797314512a7556b2bcd4af407 P eeee1578317046a7acff34fd57ab3022: No bootstrap required, opened a new log
I20260620 18:32:10.346647 13019 ts_tablet_manager.cc:1397] T 8764ebe797314512a7556b2bcd4af407 P eeee1578317046a7acff34fd57ab3022: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 18:32:10.349042 13016 raft_consensus.cc:348] T 8764ebe797314512a7556b2bcd4af407 P 38da097b32ae4db9a2d2ad6f5e03d266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } }
I20260620 18:32:10.349450 13016 raft_consensus.cc:374] T 8764ebe797314512a7556b2bcd4af407 P 38da097b32ae4db9a2d2ad6f5e03d266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:10.349556 13016 raft_consensus.cc:729] T 8764ebe797314512a7556b2bcd4af407 P 38da097b32ae4db9a2d2ad6f5e03d266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38da097b32ae4db9a2d2ad6f5e03d266, State: Initialized, Role: FOLLOWER
I20260620 18:32:10.349869 13015 raft_consensus.cc:348] T 8764ebe797314512a7556b2bcd4af407 P 5b244d023c7247219f0d39a360b3e3e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } }
I20260620 18:32:10.350258 13015 raft_consensus.cc:374] T 8764ebe797314512a7556b2bcd4af407 P 5b244d023c7247219f0d39a360b3e3e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:10.350350 13015 raft_consensus.cc:729] T 8764ebe797314512a7556b2bcd4af407 P 5b244d023c7247219f0d39a360b3e3e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b244d023c7247219f0d39a360b3e3e0, State: Initialized, Role: FOLLOWER
I20260620 18:32:10.350234 13016 consensus_queue.cc:260] T 8764ebe797314512a7556b2bcd4af407 P 38da097b32ae4db9a2d2ad6f5e03d266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } }
I20260620 18:32:10.350930 13015 consensus_queue.cc:260] T 8764ebe797314512a7556b2bcd4af407 P 5b244d023c7247219f0d39a360b3e3e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } }
I20260620 18:32:10.351984 12996 heartbeater.cc:503] Master 127.10.131.62:43781 was elected leader, sending a full tablet report...
I20260620 18:32:10.352560 13015 ts_tablet_manager.cc:1428] T 8764ebe797314512a7556b2bcd4af407 P 5b244d023c7247219f0d39a360b3e3e0: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260620 18:32:10.353332 13016 ts_tablet_manager.cc:1428] T 8764ebe797314512a7556b2bcd4af407 P 38da097b32ae4db9a2d2ad6f5e03d266: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260620 18:32:10.358711 13019 raft_consensus.cc:348] T 8764ebe797314512a7556b2bcd4af407 P eeee1578317046a7acff34fd57ab3022 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } }
I20260620 18:32:10.359153 13019 raft_consensus.cc:374] T 8764ebe797314512a7556b2bcd4af407 P eeee1578317046a7acff34fd57ab3022 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:10.359265 13019 raft_consensus.cc:729] T 8764ebe797314512a7556b2bcd4af407 P eeee1578317046a7acff34fd57ab3022 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeee1578317046a7acff34fd57ab3022, State: Initialized, Role: FOLLOWER
I20260620 18:32:10.359849 13019 consensus_queue.cc:260] T 8764ebe797314512a7556b2bcd4af407 P eeee1578317046a7acff34fd57ab3022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } }
I20260620 18:32:10.360888 12856 heartbeater.cc:503] Master 127.10.131.62:43781 was elected leader, sending a full tablet report...
I20260620 18:32:10.361479 13019 ts_tablet_manager.cc:1428] T 8764ebe797314512a7556b2bcd4af407 P eeee1578317046a7acff34fd57ab3022: Time spent starting tablet: real 0.015s user 0.008s sys 0.004s
I20260620 18:32:10.397393 13023 raft_consensus.cc:482] T 8764ebe797314512a7556b2bcd4af407 P eeee1578317046a7acff34fd57ab3022 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:32:10.397763 13023 raft_consensus.cc:504] T 8764ebe797314512a7556b2bcd4af407 P eeee1578317046a7acff34fd57ab3022 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } }
I20260620 18:32:10.399423 13023 leader_election.cc:288] T 8764ebe797314512a7556b2bcd4af407 P eeee1578317046a7acff34fd57ab3022 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38da097b32ae4db9a2d2ad6f5e03d266 (127.10.131.1:41907), 5b244d023c7247219f0d39a360b3e3e0 (127.10.131.3:35431)
W20260620 18:32:10.402475 12857 tablet.cc:2369] T 8764ebe797314512a7556b2bcd4af407 P eeee1578317046a7acff34fd57ab3022: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:32:10.408687 13011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:10.399962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:46257 (local address 127.10.131.1:41907)
0620 18:32:10.400195 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:10.400202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:10.400806 (+ 604us) server_negotiation.cc:407] Connection header received
0620 18:32:10.401066 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:10.401073 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:10.401155 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:10.401270 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:10.402161 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.403267 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.406441 (+ 3174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.407753 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.407923 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:10.408000 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:10.408376 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 18:32:10.408480 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:32:10.408787 13024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:10.399747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:41907 (local address 127.10.131.2:46257)
0620 18:32:10.400643 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:10.400665 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:10.400868 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:10.401357 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:10.401364 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:10.401399 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:10.401969 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:10.401986 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.403410 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.403414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:10.406227 (+ 2813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:10.406244 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.408056 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.408062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:10.408205 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:10.408232 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 18:32:10.408325 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 18:32:10.408432 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":165,"spinlock_wait_cycles":3456,"thread_start_us":393,"threads_started":1}
I20260620 18:32:10.409890 13012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:10.399892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:56375 (local address 127.10.131.3:35431)
0620 18:32:10.400301 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:10.400307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:10.400823 (+ 516us) server_negotiation.cc:407] Connection header received
0620 18:32:10.401066 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:10.401073 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:10.401158 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:10.401270 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:10.402367 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.403390 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.406440 (+ 3050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.407748 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.407925 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:10.408000 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:10.409456 (+ 1456us) server_negotiation.cc:299] Negotiation successful
0620 18:32:10.409639 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256}
I20260620 18:32:10.409925 13025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:10.399803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:35431 (local address 127.10.131.2:56375)
0620 18:32:10.400718 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:10.400735 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:10.400871 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:10.401457 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:10.401468 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:10.401491 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:10.402134 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:10.402149 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.403548 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.403561 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:10.406225 (+ 2664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:10.406244 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.409116 (+ 2872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.409123 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:10.409286 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:10.409310 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 18:32:10.409387 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 18:32:10.409520 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"mutex_wait_us":38,"spinlock_wait_cycles":20736,"thread_start_us":621,"threads_started":1}
I20260620 18:32:10.410120 12670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8764ebe797314512a7556b2bcd4af407" candidate_uuid: "eeee1578317046a7acff34fd57ab3022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" is_pre_election: true
I20260620 18:32:10.410797 12670 raft_consensus.cc:2454] T 8764ebe797314512a7556b2bcd4af407 P 38da097b32ae4db9a2d2ad6f5e03d266 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeee1578317046a7acff34fd57ab3022 in term 0.
I20260620 18:32:10.410899 12951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8764ebe797314512a7556b2bcd4af407" candidate_uuid: "eeee1578317046a7acff34fd57ab3022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b244d023c7247219f0d39a360b3e3e0" is_pre_election: true
I20260620 18:32:10.411442 12951 raft_consensus.cc:2454] T 8764ebe797314512a7556b2bcd4af407 P 5b244d023c7247219f0d39a360b3e3e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeee1578317046a7acff34fd57ab3022 in term 0.
I20260620 18:32:10.411549 12736 leader_election.cc:302] T 8764ebe797314512a7556b2bcd4af407 P eeee1578317046a7acff34fd57ab3022 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38da097b32ae4db9a2d2ad6f5e03d266, eeee1578317046a7acff34fd57ab3022; no voters:
I20260620 18:32:10.412027 13023 raft_consensus.cc:2781] T 8764ebe797314512a7556b2bcd4af407 P eeee1578317046a7acff34fd57ab3022 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:32:10.412217 13023 raft_consensus.cc:482] T 8764ebe797314512a7556b2bcd4af407 P eeee1578317046a7acff34fd57ab3022 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:32:10.412353 13023 raft_consensus.cc:3037] T 8764ebe797314512a7556b2bcd4af407 P eeee1578317046a7acff34fd57ab3022 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:10.414721 13023 raft_consensus.cc:504] T 8764ebe797314512a7556b2bcd4af407 P eeee1578317046a7acff34fd57ab3022 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } }
I20260620 18:32:10.415508 13023 leader_election.cc:288] T 8764ebe797314512a7556b2bcd4af407 P eeee1578317046a7acff34fd57ab3022 [CANDIDATE]: Term 1 election: Requested vote from peers 38da097b32ae4db9a2d2ad6f5e03d266 (127.10.131.1:41907), 5b244d023c7247219f0d39a360b3e3e0 (127.10.131.3:35431)
I20260620 18:32:10.415982 12670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8764ebe797314512a7556b2bcd4af407" candidate_uuid: "eeee1578317046a7acff34fd57ab3022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38da097b32ae4db9a2d2ad6f5e03d266"
I20260620 18:32:10.415982 12951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8764ebe797314512a7556b2bcd4af407" candidate_uuid: "eeee1578317046a7acff34fd57ab3022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b244d023c7247219f0d39a360b3e3e0"
I20260620 18:32:10.416210 12670 raft_consensus.cc:3037] T 8764ebe797314512a7556b2bcd4af407 P 38da097b32ae4db9a2d2ad6f5e03d266 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:10.416241 12951 raft_consensus.cc:3037] T 8764ebe797314512a7556b2bcd4af407 P 5b244d023c7247219f0d39a360b3e3e0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:10.418737 12670 raft_consensus.cc:2454] T 8764ebe797314512a7556b2bcd4af407 P 38da097b32ae4db9a2d2ad6f5e03d266 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeee1578317046a7acff34fd57ab3022 in term 1.
I20260620 18:32:10.419087 12951 raft_consensus.cc:2454] T 8764ebe797314512a7556b2bcd4af407 P 5b244d023c7247219f0d39a360b3e3e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeee1578317046a7acff34fd57ab3022 in term 1.
I20260620 18:32:10.419225 12736 leader_election.cc:302] T 8764ebe797314512a7556b2bcd4af407 P eeee1578317046a7acff34fd57ab3022 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38da097b32ae4db9a2d2ad6f5e03d266, eeee1578317046a7acff34fd57ab3022; no voters:
I20260620 18:32:10.419588 13023 raft_consensus.cc:2781] T 8764ebe797314512a7556b2bcd4af407 P eeee1578317046a7acff34fd57ab3022 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:10.420043 13023 raft_consensus.cc:686] T 8764ebe797314512a7556b2bcd4af407 P eeee1578317046a7acff34fd57ab3022 [term 1 LEADER]: Becoming Leader. State: Replica: eeee1578317046a7acff34fd57ab3022, State: Running, Role: LEADER
I20260620 18:32:10.420564 13023 consensus_queue.cc:237] T 8764ebe797314512a7556b2bcd4af407 P eeee1578317046a7acff34fd57ab3022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } }
I20260620 18:32:10.424609 12507 catalog_manager.cc:5495] T 8764ebe797314512a7556b2bcd4af407 P eeee1578317046a7acff34fd57ab3022 reported cstate change: term changed from 0 to 1, leader changed from <none> to eeee1578317046a7acff34fd57ab3022 (127.10.131.2). New cstate: current_term: 1 leader_uuid: "eeee1578317046a7acff34fd57ab3022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } health_report { overall_health: HEALTHY } } }
W20260620 18:32:10.458009 12997 tablet.cc:2369] T 8764ebe797314512a7556b2bcd4af407 P 5b244d023c7247219f0d39a360b3e3e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 18:32:10 dist-test-slave-cbn3 krb5kdc[12464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980328, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:32:10.485291 13033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:10.471359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:43781 (local address 127.0.0.1:52504)
0620 18:32:10.472024 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:10.472041 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:10.472190 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:10.472479 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:10.472482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:10.473275 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:10.473551 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:10.473560 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.474579 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.474583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:10.475531 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:10.475542 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.476315 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.476320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:10.476477 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:10.477279 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:10.477307 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:10.479474 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:10.482922 (+ 3448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:10.482931 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:10.482937 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:10.483355 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:10.483718 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:10.483721 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:10.483724 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:10.483862 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:10.484430 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:10.484436 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:10.484589 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 18:32:10.484730 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 18:32:10.484998 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":368,"threads_started":1}
I20260620 18:32:10.485558 12884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:10.471342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52504 (local address 127.10.131.62:43781)
0620 18:32:10.471516 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:10.471521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:10.472144 (+ 623us) server_negotiation.cc:407] Connection header received
0620 18:32:10.472291 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:10.472295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:10.472352 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:10.472438 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:10.473683 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.474449 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.475713 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.476160 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.476249 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:10.479633 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:10.479638 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:10.479641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:10.479667 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:10.482784 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:10.483456 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:10.483461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:10.483465 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:10.483618 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:10.483956 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:10.483961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:10.483965 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:10.484309 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:10.484443 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:10.484877 (+ 434us) server_negotiation.cc:299] Negotiation successful
0620 18:32:10.485151 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":57}
I20260620 18:32:10.490477 12507 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:52504:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 18:32:10.491545 12507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:32:10.498415 12646 tablet_service.cc:1467] Processing CreateTablet for tablet f301952af0e44c1581bfcdf3a37b93b8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f60f12454d92401b98fe74dcf51e7eb6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:32:10.498883 12926 tablet_service.cc:1467] Processing CreateTablet for tablet f301952af0e44c1581bfcdf3a37b93b8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f60f12454d92401b98fe74dcf51e7eb6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:32:10.498824 12786 tablet_service.cc:1467] Processing CreateTablet for tablet f301952af0e44c1581bfcdf3a37b93b8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f60f12454d92401b98fe74dcf51e7eb6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:32:10.499380 12646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f301952af0e44c1581bfcdf3a37b93b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:10.499677 12786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f301952af0e44c1581bfcdf3a37b93b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:10.499677 12926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f301952af0e44c1581bfcdf3a37b93b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:10.505232 13016 tablet_bootstrap.cc:492] T f301952af0e44c1581bfcdf3a37b93b8 P 38da097b32ae4db9a2d2ad6f5e03d266: Bootstrap starting.
I20260620 18:32:10.506630 13016 tablet_bootstrap.cc:654] T f301952af0e44c1581bfcdf3a37b93b8 P 38da097b32ae4db9a2d2ad6f5e03d266: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:10.507067 13015 tablet_bootstrap.cc:492] T f301952af0e44c1581bfcdf3a37b93b8 P 5b244d023c7247219f0d39a360b3e3e0: Bootstrap starting.
I20260620 18:32:10.507987 13019 tablet_bootstrap.cc:492] T f301952af0e44c1581bfcdf3a37b93b8 P eeee1578317046a7acff34fd57ab3022: Bootstrap starting.
I20260620 18:32:10.508699 13015 tablet_bootstrap.cc:654] T f301952af0e44c1581bfcdf3a37b93b8 P 5b244d023c7247219f0d39a360b3e3e0: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:10.509558 13016 tablet_bootstrap.cc:492] T f301952af0e44c1581bfcdf3a37b93b8 P 38da097b32ae4db9a2d2ad6f5e03d266: No bootstrap required, opened a new log
I20260620 18:32:10.509739 13016 ts_tablet_manager.cc:1397] T f301952af0e44c1581bfcdf3a37b93b8 P 38da097b32ae4db9a2d2ad6f5e03d266: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 18:32:10.509810 13019 tablet_bootstrap.cc:654] T f301952af0e44c1581bfcdf3a37b93b8 P eeee1578317046a7acff34fd57ab3022: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:10.510548 13016 raft_consensus.cc:348] T f301952af0e44c1581bfcdf3a37b93b8 P 38da097b32ae4db9a2d2ad6f5e03d266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } }
I20260620 18:32:10.510779 13016 raft_consensus.cc:374] T f301952af0e44c1581bfcdf3a37b93b8 P 38da097b32ae4db9a2d2ad6f5e03d266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:10.510886 13016 raft_consensus.cc:729] T f301952af0e44c1581bfcdf3a37b93b8 P 38da097b32ae4db9a2d2ad6f5e03d266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38da097b32ae4db9a2d2ad6f5e03d266, State: Initialized, Role: FOLLOWER
I20260620 18:32:10.511155 13016 consensus_queue.cc:260] T f301952af0e44c1581bfcdf3a37b93b8 P 38da097b32ae4db9a2d2ad6f5e03d266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } }
I20260620 18:32:10.511868 13016 ts_tablet_manager.cc:1428] T f301952af0e44c1581bfcdf3a37b93b8 P 38da097b32ae4db9a2d2ad6f5e03d266: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 18:32:10.511675 13021 tablet_replica.cc:439] T f301952af0e44c1581bfcdf3a37b93b8 P 38da097b32ae4db9a2d2ad6f5e03d266: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } } }
I20260620 18:32:10.512087 13021 tablet_replica.cc:442] T f301952af0e44c1581bfcdf3a37b93b8 P 38da097b32ae4db9a2d2ad6f5e03d266: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:32:10.514608 13019 tablet_bootstrap.cc:492] T f301952af0e44c1581bfcdf3a37b93b8 P eeee1578317046a7acff34fd57ab3022: No bootstrap required, opened a new log
I20260620 18:32:10.514761 13019 ts_tablet_manager.cc:1397] T f301952af0e44c1581bfcdf3a37b93b8 P eeee1578317046a7acff34fd57ab3022: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260620 18:32:10.515014 13015 tablet_bootstrap.cc:492] T f301952af0e44c1581bfcdf3a37b93b8 P 5b244d023c7247219f0d39a360b3e3e0: No bootstrap required, opened a new log
I20260620 18:32:10.515193 13015 ts_tablet_manager.cc:1397] T f301952af0e44c1581bfcdf3a37b93b8 P 5b244d023c7247219f0d39a360b3e3e0: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260620 18:32:10.515446 13019 raft_consensus.cc:348] T f301952af0e44c1581bfcdf3a37b93b8 P eeee1578317046a7acff34fd57ab3022 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } }
I20260620 18:32:10.515619 13019 raft_consensus.cc:374] T f301952af0e44c1581bfcdf3a37b93b8 P eeee1578317046a7acff34fd57ab3022 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:10.515733 13019 raft_consensus.cc:729] T f301952af0e44c1581bfcdf3a37b93b8 P eeee1578317046a7acff34fd57ab3022 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeee1578317046a7acff34fd57ab3022, State: Initialized, Role: FOLLOWER
I20260620 18:32:10.515952 13019 consensus_queue.cc:260] T f301952af0e44c1581bfcdf3a37b93b8 P eeee1578317046a7acff34fd57ab3022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } }
I20260620 18:32:10.516019 13015 raft_consensus.cc:348] T f301952af0e44c1581bfcdf3a37b93b8 P 5b244d023c7247219f0d39a360b3e3e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } }
I20260620 18:32:10.516297 13015 raft_consensus.cc:374] T f301952af0e44c1581bfcdf3a37b93b8 P 5b244d023c7247219f0d39a360b3e3e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:10.516396 13015 raft_consensus.cc:729] T f301952af0e44c1581bfcdf3a37b93b8 P 5b244d023c7247219f0d39a360b3e3e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b244d023c7247219f0d39a360b3e3e0, State: Initialized, Role: FOLLOWER
I20260620 18:32:10.516321 13023 tablet_replica.cc:439] T f301952af0e44c1581bfcdf3a37b93b8 P eeee1578317046a7acff34fd57ab3022: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } } }
I20260620 18:32:10.516602 13023 tablet_replica.cc:442] T f301952af0e44c1581bfcdf3a37b93b8 P eeee1578317046a7acff34fd57ab3022: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:32:10.516601 13019 ts_tablet_manager.cc:1428] T f301952af0e44c1581bfcdf3a37b93b8 P eeee1578317046a7acff34fd57ab3022: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 18:32:10.516633 13015 consensus_queue.cc:260] T f301952af0e44c1581bfcdf3a37b93b8 P 5b244d023c7247219f0d39a360b3e3e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } }
I20260620 18:32:10.517366 13015 ts_tablet_manager.cc:1428] T f301952af0e44c1581bfcdf3a37b93b8 P 5b244d023c7247219f0d39a360b3e3e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:32:10.517213 13022 tablet_replica.cc:439] T f301952af0e44c1581bfcdf3a37b93b8 P 5b244d023c7247219f0d39a360b3e3e0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } } }
I20260620 18:32:10.517597 13022 tablet_replica.cc:442] T f301952af0e44c1581bfcdf3a37b93b8 P 5b244d023c7247219f0d39a360b3e3e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:32:10.525334 13023 raft_consensus.cc:482] T f301952af0e44c1581bfcdf3a37b93b8 P eeee1578317046a7acff34fd57ab3022 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:32:10.525485 13023 raft_consensus.cc:504] T f301952af0e44c1581bfcdf3a37b93b8 P eeee1578317046a7acff34fd57ab3022 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } }
I20260620 18:32:10.526136 13023 leader_election.cc:288] T f301952af0e44c1581bfcdf3a37b93b8 P eeee1578317046a7acff34fd57ab3022 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38da097b32ae4db9a2d2ad6f5e03d266 (127.10.131.1:41907), 5b244d023c7247219f0d39a360b3e3e0 (127.10.131.3:35431)
I20260620 18:32:10.526571 12951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f301952af0e44c1581bfcdf3a37b93b8" candidate_uuid: "eeee1578317046a7acff34fd57ab3022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b244d023c7247219f0d39a360b3e3e0" is_pre_election: true
I20260620 18:32:10.526577 12670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f301952af0e44c1581bfcdf3a37b93b8" candidate_uuid: "eeee1578317046a7acff34fd57ab3022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" is_pre_election: true
I20260620 18:32:10.526822 12951 raft_consensus.cc:2454] T f301952af0e44c1581bfcdf3a37b93b8 P 5b244d023c7247219f0d39a360b3e3e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeee1578317046a7acff34fd57ab3022 in term 0.
I20260620 18:32:10.526822 12670 raft_consensus.cc:2454] T f301952af0e44c1581bfcdf3a37b93b8 P 38da097b32ae4db9a2d2ad6f5e03d266 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeee1578317046a7acff34fd57ab3022 in term 0.
I20260620 18:32:10.527242 12736 leader_election.cc:302] T f301952af0e44c1581bfcdf3a37b93b8 P eeee1578317046a7acff34fd57ab3022 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 38da097b32ae4db9a2d2ad6f5e03d266, 5b244d023c7247219f0d39a360b3e3e0, eeee1578317046a7acff34fd57ab3022; no voters:
I20260620 18:32:10.527606 13023 raft_consensus.cc:2781] T f301952af0e44c1581bfcdf3a37b93b8 P eeee1578317046a7acff34fd57ab3022 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:32:10.527700 13023 raft_consensus.cc:482] T f301952af0e44c1581bfcdf3a37b93b8 P eeee1578317046a7acff34fd57ab3022 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:32:10.527863 13023 raft_consensus.cc:3037] T f301952af0e44c1581bfcdf3a37b93b8 P eeee1578317046a7acff34fd57ab3022 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:10.530332 13023 raft_consensus.cc:504] T f301952af0e44c1581bfcdf3a37b93b8 P eeee1578317046a7acff34fd57ab3022 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } }
I20260620 18:32:10.531004 13023 leader_election.cc:288] T f301952af0e44c1581bfcdf3a37b93b8 P eeee1578317046a7acff34fd57ab3022 [CANDIDATE]: Term 1 election: Requested vote from peers 38da097b32ae4db9a2d2ad6f5e03d266 (127.10.131.1:41907), 5b244d023c7247219f0d39a360b3e3e0 (127.10.131.3:35431)
I20260620 18:32:10.531461 12951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f301952af0e44c1581bfcdf3a37b93b8" candidate_uuid: "eeee1578317046a7acff34fd57ab3022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b244d023c7247219f0d39a360b3e3e0"
I20260620 18:32:10.531450 12670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f301952af0e44c1581bfcdf3a37b93b8" candidate_uuid: "eeee1578317046a7acff34fd57ab3022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38da097b32ae4db9a2d2ad6f5e03d266"
I20260620 18:32:10.531677 12951 raft_consensus.cc:3037] T f301952af0e44c1581bfcdf3a37b93b8 P 5b244d023c7247219f0d39a360b3e3e0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:10.531697 12670 raft_consensus.cc:3037] T f301952af0e44c1581bfcdf3a37b93b8 P 38da097b32ae4db9a2d2ad6f5e03d266 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:10.534673 12951 raft_consensus.cc:2454] T f301952af0e44c1581bfcdf3a37b93b8 P 5b244d023c7247219f0d39a360b3e3e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeee1578317046a7acff34fd57ab3022 in term 1.
I20260620 18:32:10.534673 12670 raft_consensus.cc:2454] T f301952af0e44c1581bfcdf3a37b93b8 P 38da097b32ae4db9a2d2ad6f5e03d266 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeee1578317046a7acff34fd57ab3022 in term 1.
I20260620 18:32:10.535061 12736 leader_election.cc:302] T f301952af0e44c1581bfcdf3a37b93b8 P eeee1578317046a7acff34fd57ab3022 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 38da097b32ae4db9a2d2ad6f5e03d266, 5b244d023c7247219f0d39a360b3e3e0, eeee1578317046a7acff34fd57ab3022; no voters:
I20260620 18:32:10.535411 13023 raft_consensus.cc:2781] T f301952af0e44c1581bfcdf3a37b93b8 P eeee1578317046a7acff34fd57ab3022 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:10.535591 13023 raft_consensus.cc:686] T f301952af0e44c1581bfcdf3a37b93b8 P eeee1578317046a7acff34fd57ab3022 [term 1 LEADER]: Becoming Leader. State: Replica: eeee1578317046a7acff34fd57ab3022, State: Running, Role: LEADER
I20260620 18:32:10.535847 13023 consensus_queue.cc:237] T f301952af0e44c1581bfcdf3a37b93b8 P eeee1578317046a7acff34fd57ab3022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } }
I20260620 18:32:10.537426 13026 tablet_replica.cc:439] T f301952af0e44c1581bfcdf3a37b93b8 P eeee1578317046a7acff34fd57ab3022: TxnStatusTablet state changed. Reason: New leader eeee1578317046a7acff34fd57ab3022. Latest consensus state: current_term: 1 leader_uuid: "eeee1578317046a7acff34fd57ab3022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } } }
I20260620 18:32:10.537771 13026 tablet_replica.cc:442] T f301952af0e44c1581bfcdf3a37b93b8 P eeee1578317046a7acff34fd57ab3022: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:32:10.538480 13038 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 18:32:10.538758 13038 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 18:32:10.538986 12509 catalog_manager.cc:5495] T f301952af0e44c1581bfcdf3a37b93b8 P eeee1578317046a7acff34fd57ab3022 reported cstate change: term changed from 0 to 1, leader changed from <none> to eeee1578317046a7acff34fd57ab3022 (127.10.131.2). New cstate: current_term: 1 leader_uuid: "eeee1578317046a7acff34fd57ab3022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } health_report { overall_health: UNKNOWN } } }
I20260620 18:32:10.565215 13033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:10.558373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:36507 (local address 127.0.0.1:50924)
0620 18:32:10.558644 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:10.558675 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:10.558824 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:10.559361 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:10.559368 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:10.559384 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:32:10.559593 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:10.559604 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.560673 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.560679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:10.562291 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:10.562306 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.563338 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.563346 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:10.563553 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:10.563628 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:32:10.564580 (+ 952us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:32:10.564602 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:32:10.564715 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 18:32:10.564908 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 18:32:10.565341 13014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:10.558421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50924 (local address 127.10.131.2:36507)
0620 18:32:10.558627 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:10.558635 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:10.558754 (+ 119us) server_negotiation.cc:407] Connection header received
0620 18:32:10.559075 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:10.559079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:10.559171 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:10.559299 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:32:10.559777 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.560514 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.562487 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.563169 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.563281 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:10.563879 (+ 598us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:32:10.564455 (+ 576us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:32:10.564579 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:10.564746 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 18:32:10.564823 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:32:10.573616 12951 raft_consensus.cc:1261] T f301952af0e44c1581bfcdf3a37b93b8 P 5b244d023c7247219f0d39a360b3e3e0 [term 1 FOLLOWER]: Refusing update from remote peer eeee1578317046a7acff34fd57ab3022: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:10.574529 12670 raft_consensus.cc:1261] T f301952af0e44c1581bfcdf3a37b93b8 P 38da097b32ae4db9a2d2ad6f5e03d266 [term 1 FOLLOWER]: Refusing update from remote peer eeee1578317046a7acff34fd57ab3022: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:10.574570 13026 consensus_queue.cc:1035] T f301952af0e44c1581bfcdf3a37b93b8 P eeee1578317046a7acff34fd57ab3022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:10.575349 13026 consensus_queue.cc:1035] T f301952af0e44c1581bfcdf3a37b93b8 P eeee1578317046a7acff34fd57ab3022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 18:32:10.587064 12717 tablet.cc:2369] T 8764ebe797314512a7556b2bcd4af407 P 38da097b32ae4db9a2d2ad6f5e03d266: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:32:10.589111 13022 tablet_replica.cc:439] T f301952af0e44c1581bfcdf3a37b93b8 P 5b244d023c7247219f0d39a360b3e3e0: TxnStatusTablet state changed. Reason: New leader eeee1578317046a7acff34fd57ab3022. Latest consensus state: current_term: 1 leader_uuid: "eeee1578317046a7acff34fd57ab3022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } } }
I20260620 18:32:10.589404 13022 tablet_replica.cc:442] T f301952af0e44c1581bfcdf3a37b93b8 P 5b244d023c7247219f0d39a360b3e3e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:32:10.591293 13041 mvcc.cc:205] Tried to move back new op lower bound from 7298991434020233216 to 7298991433876692992. Current Snapshot: MvccSnapshot[applied={T|T < 7298991434020233216}]
I20260620 18:32:10.594928 13021 tablet_replica.cc:439] T f301952af0e44c1581bfcdf3a37b93b8 P 38da097b32ae4db9a2d2ad6f5e03d266: TxnStatusTablet state changed. Reason: New leader eeee1578317046a7acff34fd57ab3022. Latest consensus state: current_term: 1 leader_uuid: "eeee1578317046a7acff34fd57ab3022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } } }
I20260620 18:32:10.595213 13021 tablet_replica.cc:442] T f301952af0e44c1581bfcdf3a37b93b8 P 38da097b32ae4db9a2d2ad6f5e03d266: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:32:10.599704 13023 tablet_replica.cc:439] T f301952af0e44c1581bfcdf3a37b93b8 P eeee1578317046a7acff34fd57ab3022: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eeee1578317046a7acff34fd57ab3022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } } }
I20260620 18:32:10.600004 13023 tablet_replica.cc:442] T f301952af0e44c1581bfcdf3a37b93b8 P eeee1578317046a7acff34fd57ab3022: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:32:10.600404 13043 mvcc.cc:205] Tried to move back new op lower bound from 7298991434020233216 to 7298991433876692992. Current Snapshot: MvccSnapshot[applied={T|T < 7298991434020233216}]
I20260620 18:32:10.603533 13023 tablet_replica.cc:439] T f301952af0e44c1581bfcdf3a37b93b8 P eeee1578317046a7acff34fd57ab3022: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eeee1578317046a7acff34fd57ab3022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } } }
I20260620 18:32:10.603858 13023 tablet_replica.cc:442] T f301952af0e44c1581bfcdf3a37b93b8 P eeee1578317046a7acff34fd57ab3022: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:32:10.607828 13022 tablet_replica.cc:439] T f301952af0e44c1581bfcdf3a37b93b8 P 5b244d023c7247219f0d39a360b3e3e0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eeee1578317046a7acff34fd57ab3022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } } }
I20260620 18:32:10.608140 13022 tablet_replica.cc:442] T f301952af0e44c1581bfcdf3a37b93b8 P 5b244d023c7247219f0d39a360b3e3e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:32:10.609452 13021 tablet_replica.cc:439] T f301952af0e44c1581bfcdf3a37b93b8 P 38da097b32ae4db9a2d2ad6f5e03d266: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eeee1578317046a7acff34fd57ab3022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } } }
I20260620 18:32:10.609704 13021 tablet_replica.cc:442] T f301952af0e44c1581bfcdf3a37b93b8 P 38da097b32ae4db9a2d2ad6f5e03d266: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:32:10.619163 13014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:10.613367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50934 (local address 127.10.131.2:36507)
0620 18:32:10.613559 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:10.613563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:10.613611 (+ 48us) server_negotiation.cc:407] Connection header received
0620 18:32:10.613839 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:10.613842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:10.613951 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:10.614022 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:32:10.614417 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.615191 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.617260 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.617685 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.617780 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:10.618353 (+ 573us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:32:10.618503 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:32:10.618594 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:10.618757 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 18:32:10.618848 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260620 18:32:10.638229 13014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:10.631465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:36325 (local address 127.10.131.2:36507)
0620 18:32:10.631708 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:10.631715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:10.632384 (+ 669us) server_negotiation.cc:407] Connection header received
0620 18:32:10.632524 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:10.632529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:10.632619 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:10.632729 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:32:10.633460 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.634627 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.636102 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.636786 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.636960 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:10.637460 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:32:10.637598 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:32:10.637699 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:10.637821 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 18:32:10.637898 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:32:10.638229 13052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:10.631465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:36507 (local address 127.10.131.2:36325)
0620 18:32:10.632274 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:10.632296 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:10.632422 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:10.632764 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:10.632767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:10.632781 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:32:10.633305 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:10.633315 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.634780 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.634784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:10.635961 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:10.635973 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.637011 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.637016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:10.637186 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:10.637246 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:32:10.637706 (+ 460us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:32:10.637715 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 18:32:10.637777 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 18:32:10.637876 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":423,"threads_started":1}
I20260620 18:32:10.648094 12951 raft_consensus.cc:1261] T 8764ebe797314512a7556b2bcd4af407 P 5b244d023c7247219f0d39a360b3e3e0 [term 1 FOLLOWER]: Refusing update from remote peer eeee1578317046a7acff34fd57ab3022: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:10.648175 12669 raft_consensus.cc:1261] T 8764ebe797314512a7556b2bcd4af407 P 38da097b32ae4db9a2d2ad6f5e03d266 [term 1 FOLLOWER]: Refusing update from remote peer eeee1578317046a7acff34fd57ab3022: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:10.648797 13026 consensus_queue.cc:1035] T 8764ebe797314512a7556b2bcd4af407 P eeee1578317046a7acff34fd57ab3022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:10.649410 13026 consensus_queue.cc:1035] T 8764ebe797314512a7556b2bcd4af407 P eeee1578317046a7acff34fd57ab3022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:10.727349 12507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:32:10.727653 12507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:32:10.731200 12507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f77c29104bae4b95b9f3c560d0589bb9: Sending DeleteTablet for 3 replicas of tablet 8764ebe797314512a7556b2bcd4af407
I20260620 18:32:10.732257 12646 tablet_service.cc:1514] Processing DeleteTablet for tablet 8764ebe797314512a7556b2bcd4af407 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:10 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:60390
I20260620 18:32:10.732503 12785 tablet_service.cc:1514] Processing DeleteTablet for tablet 8764ebe797314512a7556b2bcd4af407 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:10 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:50920
I20260620 18:32:10.732982 12926 tablet_service.cc:1514] Processing DeleteTablet for tablet 8764ebe797314512a7556b2bcd4af407 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:10 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:59370
I20260620 18:32:10.733500 13058 tablet_replica.cc:330] T 8764ebe797314512a7556b2bcd4af407 P 38da097b32ae4db9a2d2ad6f5e03d266: stopping tablet replica
I20260620 18:32:10.734086 13058 raft_consensus.cc:2229] T 8764ebe797314512a7556b2bcd4af407 P 38da097b32ae4db9a2d2ad6f5e03d266 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:32:10.734573 13058 raft_consensus.cc:2258] T 8764ebe797314512a7556b2bcd4af407 P 38da097b32ae4db9a2d2ad6f5e03d266 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:10.735550 12507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:32:10.736588 12507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:32:10.742357 13058 ts_tablet_manager.cc:1905] T 8764ebe797314512a7556b2bcd4af407 P 38da097b32ae4db9a2d2ad6f5e03d266: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:10.744473 12786 tablet_service.cc:1467] Processing CreateTablet for tablet 1a887236b132491db2f6a232baef1e0c (DEFAULT_TABLE table=test-table [id=48fd067c74c54e11b862ca5984f8a564]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:10.746089 12925 tablet_service.cc:1467] Processing CreateTablet for tablet 1a887236b132491db2f6a232baef1e0c (DEFAULT_TABLE table=test-table [id=48fd067c74c54e11b862ca5984f8a564]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:10.746390 12646 tablet_service.cc:1467] Processing CreateTablet for tablet 1a887236b132491db2f6a232baef1e0c (DEFAULT_TABLE table=test-table [id=48fd067c74c54e11b862ca5984f8a564]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:10.746537 12925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a887236b132491db2f6a232baef1e0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:10.746913 12646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a887236b132491db2f6a232baef1e0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:10.749626 12786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a887236b132491db2f6a232baef1e0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:10.753247 13015 tablet_bootstrap.cc:492] T 1a887236b132491db2f6a232baef1e0c P 5b244d023c7247219f0d39a360b3e3e0: Bootstrap starting.
I20260620 18:32:10.755452 13015 tablet_bootstrap.cc:654] T 1a887236b132491db2f6a232baef1e0c P 5b244d023c7247219f0d39a360b3e3e0: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:10.757051 13060 tablet_replica.cc:330] T 8764ebe797314512a7556b2bcd4af407 P 5b244d023c7247219f0d39a360b3e3e0: stopping tablet replica
I20260620 18:32:10.757665 13060 raft_consensus.cc:2229] T 8764ebe797314512a7556b2bcd4af407 P 5b244d023c7247219f0d39a360b3e3e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:32:10.758086 13060 raft_consensus.cc:2258] T 8764ebe797314512a7556b2bcd4af407 P 5b244d023c7247219f0d39a360b3e3e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:10.760213 13060 ts_tablet_manager.cc:1905] T 8764ebe797314512a7556b2bcd4af407 P 5b244d023c7247219f0d39a360b3e3e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:10.763012 13059 tablet_replica.cc:330] T 8764ebe797314512a7556b2bcd4af407 P eeee1578317046a7acff34fd57ab3022: stopping tablet replica
I20260620 18:32:10.763461 13059 raft_consensus.cc:2229] T 8764ebe797314512a7556b2bcd4af407 P eeee1578317046a7acff34fd57ab3022 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:32:10.764226 13059 raft_consensus.cc:2258] T 8764ebe797314512a7556b2bcd4af407 P eeee1578317046a7acff34fd57ab3022 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:10.767603 13016 tablet_bootstrap.cc:492] T 1a887236b132491db2f6a232baef1e0c P 38da097b32ae4db9a2d2ad6f5e03d266: Bootstrap starting.
I20260620 18:32:10.769476 13016 tablet_bootstrap.cc:654] T 1a887236b132491db2f6a232baef1e0c P 38da097b32ae4db9a2d2ad6f5e03d266: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:10.769687 13060 ts_tablet_manager.cc:1918] T 8764ebe797314512a7556b2bcd4af407 P 5b244d023c7247219f0d39a360b3e3e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:32:10.769878 13060 log.cc:1196] T 8764ebe797314512a7556b2bcd4af407 P 5b244d023c7247219f0d39a360b3e3e0: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-2/wal/wals/8764ebe797314512a7556b2bcd4af407
I20260620 18:32:10.770489 13060 ts_tablet_manager.cc:1939] T 8764ebe797314512a7556b2bcd4af407 P 5b244d023c7247219f0d39a360b3e3e0: Deleting consensus metadata
I20260620 18:32:10.772724 12494 catalog_manager.cc:4841] TS 5b244d023c7247219f0d39a360b3e3e0 (127.10.131.3:35431): tablet 8764ebe797314512a7556b2bcd4af407 (table test-table [id=bf9e1bda58a54ff59399eb4d84f05b54]) successfully deleted
I20260620 18:32:10.773144 13019 tablet_bootstrap.cc:492] T 1a887236b132491db2f6a232baef1e0c P eeee1578317046a7acff34fd57ab3022: Bootstrap starting.
I20260620 18:32:10.775210 13059 ts_tablet_manager.cc:1905] T 8764ebe797314512a7556b2bcd4af407 P eeee1578317046a7acff34fd57ab3022: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:10.775291 13019 tablet_bootstrap.cc:654] T 1a887236b132491db2f6a232baef1e0c P eeee1578317046a7acff34fd57ab3022: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:10.783542 13059 ts_tablet_manager.cc:1918] T 8764ebe797314512a7556b2bcd4af407 P eeee1578317046a7acff34fd57ab3022: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:32:10.783710 13059 log.cc:1196] T 8764ebe797314512a7556b2bcd4af407 P eeee1578317046a7acff34fd57ab3022: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-1/wal/wals/8764ebe797314512a7556b2bcd4af407
I20260620 18:32:10.784092 13019 tablet_bootstrap.cc:492] T 1a887236b132491db2f6a232baef1e0c P eeee1578317046a7acff34fd57ab3022: No bootstrap required, opened a new log
I20260620 18:32:10.784243 13059 ts_tablet_manager.cc:1939] T 8764ebe797314512a7556b2bcd4af407 P eeee1578317046a7acff34fd57ab3022: Deleting consensus metadata
I20260620 18:32:10.784297 13019 ts_tablet_manager.cc:1397] T 1a887236b132491db2f6a232baef1e0c P eeee1578317046a7acff34fd57ab3022: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260620 18:32:10.785341 13019 raft_consensus.cc:348] T 1a887236b132491db2f6a232baef1e0c P eeee1578317046a7acff34fd57ab3022 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } }
I20260620 18:32:10.785661 13019 raft_consensus.cc:374] T 1a887236b132491db2f6a232baef1e0c P eeee1578317046a7acff34fd57ab3022 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:10.785751 13019 raft_consensus.cc:729] T 1a887236b132491db2f6a232baef1e0c P eeee1578317046a7acff34fd57ab3022 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeee1578317046a7acff34fd57ab3022, State: Initialized, Role: FOLLOWER
I20260620 18:32:10.786015 13019 consensus_queue.cc:260] T 1a887236b132491db2f6a232baef1e0c P eeee1578317046a7acff34fd57ab3022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } }
I20260620 18:32:10.786412 12494 catalog_manager.cc:4841] TS eeee1578317046a7acff34fd57ab3022 (127.10.131.2:36507): tablet 8764ebe797314512a7556b2bcd4af407 (table test-table [id=bf9e1bda58a54ff59399eb4d84f05b54]) successfully deleted
I20260620 18:32:10.786581 13019 ts_tablet_manager.cc:1428] T 1a887236b132491db2f6a232baef1e0c P eeee1578317046a7acff34fd57ab3022: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:32:10.789117 13058 ts_tablet_manager.cc:1918] T 8764ebe797314512a7556b2bcd4af407 P 38da097b32ae4db9a2d2ad6f5e03d266: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:32:10.789316 13058 log.cc:1196] T 8764ebe797314512a7556b2bcd4af407 P 38da097b32ae4db9a2d2ad6f5e03d266: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-0/wal/wals/8764ebe797314512a7556b2bcd4af407
I20260620 18:32:10.789867 13058 ts_tablet_manager.cc:1939] T 8764ebe797314512a7556b2bcd4af407 P 38da097b32ae4db9a2d2ad6f5e03d266: Deleting consensus metadata
I20260620 18:32:10.790854 13016 tablet_bootstrap.cc:492] T 1a887236b132491db2f6a232baef1e0c P 38da097b32ae4db9a2d2ad6f5e03d266: No bootstrap required, opened a new log
I20260620 18:32:10.791206 13015 tablet_bootstrap.cc:492] T 1a887236b132491db2f6a232baef1e0c P 5b244d023c7247219f0d39a360b3e3e0: No bootstrap required, opened a new log
I20260620 18:32:10.791167 13016 ts_tablet_manager.cc:1397] T 1a887236b132491db2f6a232baef1e0c P 38da097b32ae4db9a2d2ad6f5e03d266: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.010s
I20260620 18:32:10.791374 13015 ts_tablet_manager.cc:1397] T 1a887236b132491db2f6a232baef1e0c P 5b244d023c7247219f0d39a360b3e3e0: Time spent bootstrapping tablet: real 0.038s user 0.008s sys 0.014s
I20260620 18:32:10.791719 12493 catalog_manager.cc:4841] TS 38da097b32ae4db9a2d2ad6f5e03d266 (127.10.131.1:41907): tablet 8764ebe797314512a7556b2bcd4af407 (table test-table [id=bf9e1bda58a54ff59399eb4d84f05b54]) successfully deleted
I20260620 18:32:10.792132 13016 raft_consensus.cc:348] T 1a887236b132491db2f6a232baef1e0c P 38da097b32ae4db9a2d2ad6f5e03d266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } }
I20260620 18:32:10.792110 13015 raft_consensus.cc:348] T 1a887236b132491db2f6a232baef1e0c P 5b244d023c7247219f0d39a360b3e3e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } }
I20260620 18:32:10.792353 13015 raft_consensus.cc:374] T 1a887236b132491db2f6a232baef1e0c P 5b244d023c7247219f0d39a360b3e3e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:10.792361 13016 raft_consensus.cc:374] T 1a887236b132491db2f6a232baef1e0c P 38da097b32ae4db9a2d2ad6f5e03d266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:10.792436 13016 raft_consensus.cc:729] T 1a887236b132491db2f6a232baef1e0c P 38da097b32ae4db9a2d2ad6f5e03d266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38da097b32ae4db9a2d2ad6f5e03d266, State: Initialized, Role: FOLLOWER
I20260620 18:32:10.792434 13015 raft_consensus.cc:729] T 1a887236b132491db2f6a232baef1e0c P 5b244d023c7247219f0d39a360b3e3e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b244d023c7247219f0d39a360b3e3e0, State: Initialized, Role: FOLLOWER
I20260620 18:32:10.792665 13015 consensus_queue.cc:260] T 1a887236b132491db2f6a232baef1e0c P 5b244d023c7247219f0d39a360b3e3e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } }
I20260620 18:32:10.792682 13016 consensus_queue.cc:260] T 1a887236b132491db2f6a232baef1e0c P 38da097b32ae4db9a2d2ad6f5e03d266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } }
I20260620 18:32:10.793285 13015 ts_tablet_manager.cc:1428] T 1a887236b132491db2f6a232baef1e0c P 5b244d023c7247219f0d39a360b3e3e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:32:10.793366 13016 ts_tablet_manager.cc:1428] T 1a887236b132491db2f6a232baef1e0c P 38da097b32ae4db9a2d2ad6f5e03d266: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:32:10.945477 13022 raft_consensus.cc:482] T 1a887236b132491db2f6a232baef1e0c P 5b244d023c7247219f0d39a360b3e3e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:32:10.945752 13022 raft_consensus.cc:504] T 1a887236b132491db2f6a232baef1e0c P 5b244d023c7247219f0d39a360b3e3e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } }
I20260620 18:32:10.947846 13022 leader_election.cc:288] T 1a887236b132491db2f6a232baef1e0c P 5b244d023c7247219f0d39a360b3e3e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38da097b32ae4db9a2d2ad6f5e03d266 (127.10.131.1:41907), eeee1578317046a7acff34fd57ab3022 (127.10.131.2:36507)
I20260620 18:32:10.957718 13064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:10.948077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:41907 (local address 127.10.131.3:51923)
0620 18:32:10.948580 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:10.948604 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:10.948765 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:10.949394 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:10.949399 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:10.949427 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:10.949663 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:10.949675 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.951599 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.951607 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:10.954571 (+ 2964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:10.954591 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.956676 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.956686 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:10.956947 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:10.956996 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 18:32:10.957100 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 18:32:10.957288 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":1802,"threads_started":1}
I20260620 18:32:10.958664 13065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:10.948521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:51923 (local address 127.10.131.1:41907)
0620 18:32:10.949052 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:10.949056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:10.949071 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:32:10.949131 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:10.949135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:10.949205 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:10.949331 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:10.949860 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.951393 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.954779 (+ 3386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.956273 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.956465 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:10.956577 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:10.958014 (+ 1437us) server_negotiation.cc:299] Negotiation successful
0620 18:32:10.958224 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":1200,"threads_started":1}
I20260620 18:32:10.959437 12669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a887236b132491db2f6a232baef1e0c" candidate_uuid: "5b244d023c7247219f0d39a360b3e3e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" is_pre_election: true
I20260620 18:32:10.959790 12669 raft_consensus.cc:2454] T 1a887236b132491db2f6a232baef1e0c P 38da097b32ae4db9a2d2ad6f5e03d266 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b244d023c7247219f0d39a360b3e3e0 in term 0.
I20260620 18:32:10.960350 12876 leader_election.cc:302] T 1a887236b132491db2f6a232baef1e0c P 5b244d023c7247219f0d39a360b3e3e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38da097b32ae4db9a2d2ad6f5e03d266, 5b244d023c7247219f0d39a360b3e3e0; no voters:
I20260620 18:32:10.960747 13022 raft_consensus.cc:2781] T 1a887236b132491db2f6a232baef1e0c P 5b244d023c7247219f0d39a360b3e3e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:32:10.960932 13022 raft_consensus.cc:482] T 1a887236b132491db2f6a232baef1e0c P 5b244d023c7247219f0d39a360b3e3e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:32:10.961058 13022 raft_consensus.cc:3037] T 1a887236b132491db2f6a232baef1e0c P 5b244d023c7247219f0d39a360b3e3e0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:10.963954 13022 raft_consensus.cc:504] T 1a887236b132491db2f6a232baef1e0c P 5b244d023c7247219f0d39a360b3e3e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } }
I20260620 18:32:10.964632 13022 leader_election.cc:288] T 1a887236b132491db2f6a232baef1e0c P 5b244d023c7247219f0d39a360b3e3e0 [CANDIDATE]: Term 1 election: Requested vote from peers 38da097b32ae4db9a2d2ad6f5e03d266 (127.10.131.1:41907), eeee1578317046a7acff34fd57ab3022 (127.10.131.2:36507)
I20260620 18:32:10.965116 12669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a887236b132491db2f6a232baef1e0c" candidate_uuid: "5b244d023c7247219f0d39a360b3e3e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38da097b32ae4db9a2d2ad6f5e03d266"
I20260620 18:32:10.965387 12669 raft_consensus.cc:3037] T 1a887236b132491db2f6a232baef1e0c P 38da097b32ae4db9a2d2ad6f5e03d266 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:10.967460 12669 raft_consensus.cc:2454] T 1a887236b132491db2f6a232baef1e0c P 38da097b32ae4db9a2d2ad6f5e03d266 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b244d023c7247219f0d39a360b3e3e0 in term 1.
I20260620 18:32:10.967717 13066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:10.949094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:36507 (local address 127.10.131.3:42613)
0620 18:32:10.959350 (+ 10256us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:10.959372 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:10.959569 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:10.960349 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:10.960354 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:10.960380 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:10.960824 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:10.960868 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.962370 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.962376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:10.964867 (+ 2491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:10.964885 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.966897 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.966902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:10.967074 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:10.967111 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 18:32:10.967217 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 18:32:10.967376 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9382,"thread_start_us":9408,"threads_started":1}
I20260620 18:32:10.967913 12876 leader_election.cc:302] T 1a887236b132491db2f6a232baef1e0c P 5b244d023c7247219f0d39a360b3e3e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38da097b32ae4db9a2d2ad6f5e03d266, 5b244d023c7247219f0d39a360b3e3e0; no voters:
I20260620 18:32:10.968086 13014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:10.949498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:42613 (local address 127.10.131.2:36507)
0620 18:32:10.949705 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:10.949711 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:10.959458 (+ 9747us) server_negotiation.cc:407] Connection header received
0620 18:32:10.959863 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:10.959870 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:10.959956 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:10.960028 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:10.961065 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.962191 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.965107 (+ 2916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:10.966512 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:10.966652 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:10.966728 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:10.967316 (+ 588us) server_negotiation.cc:299] Negotiation successful
0620 18:32:10.967467 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 18:32:10.969043 13022 raft_consensus.cc:2781] T 1a887236b132491db2f6a232baef1e0c P 5b244d023c7247219f0d39a360b3e3e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:10.968899 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a887236b132491db2f6a232baef1e0c" candidate_uuid: "5b244d023c7247219f0d39a360b3e3e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeee1578317046a7acff34fd57ab3022" is_pre_election: true
I20260620 18:32:10.969017 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a887236b132491db2f6a232baef1e0c" candidate_uuid: "5b244d023c7247219f0d39a360b3e3e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeee1578317046a7acff34fd57ab3022"
I20260620 18:32:10.969380 12806 raft_consensus.cc:2454] T 1a887236b132491db2f6a232baef1e0c P eeee1578317046a7acff34fd57ab3022 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b244d023c7247219f0d39a360b3e3e0 in term 0.
I20260620 18:32:10.973145 13022 raft_consensus.cc:686] T 1a887236b132491db2f6a232baef1e0c P 5b244d023c7247219f0d39a360b3e3e0 [term 1 LEADER]: Becoming Leader. State: Replica: 5b244d023c7247219f0d39a360b3e3e0, State: Running, Role: LEADER
I20260620 18:32:10.973562 13022 consensus_queue.cc:237] T 1a887236b132491db2f6a232baef1e0c P 5b244d023c7247219f0d39a360b3e3e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } }
I20260620 18:32:10.977449 12509 catalog_manager.cc:5495] T 1a887236b132491db2f6a232baef1e0c P 5b244d023c7247219f0d39a360b3e3e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b244d023c7247219f0d39a360b3e3e0 (127.10.131.3). New cstate: current_term: 1 leader_uuid: "5b244d023c7247219f0d39a360b3e3e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } health_report { overall_health: HEALTHY } } }
I20260620 18:32:11.019668 13069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:11.013548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59386 (local address 127.10.131.3:35431)
0620 18:32:11.014220 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:11.014228 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:11.014253 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:32:11.014371 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:11.014378 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:11.014456 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:11.014627 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:32:11.015088 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:11.015821 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:11.017208 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:11.017673 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:11.017821 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:11.018377 (+ 556us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:32:11.018996 (+ 619us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:32:11.019100 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:11.019243 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 18:32:11.019306 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":379,"threads_started":1}
I20260620 18:32:11.037168 13072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:11.030628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:36507 (local address 127.10.131.3:37149)
0620 18:32:11.031343 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:11.031359 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:11.031472 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:11.031938 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:11.031942 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:11.031951 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:32:11.032193 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:11.032200 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:11.033790 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:11.033794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:11.034831 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:11.034843 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:11.035945 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:11.035949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:11.036079 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:11.036136 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:32:11.036644 (+ 508us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:32:11.036653 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 18:32:11.036726 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 18:32:11.036829 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":378,"threads_started":1}
I20260620 18:32:11.037173 13014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:11.030681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:37149 (local address 127.10.131.2:36507)
0620 18:32:11.030906 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:11.030913 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:11.031451 (+ 538us) server_negotiation.cc:407] Connection header received
0620 18:32:11.031614 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:11.031630 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:11.031744 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:11.031864 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:32:11.032360 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:11.033660 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:11.034979 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:11.035757 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:11.035897 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:11.036353 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:32:11.036535 (+ 182us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:32:11.036641 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:11.036773 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 18:32:11.036879 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 18:32:11.046394 12805 raft_consensus.cc:3037] T 1a887236b132491db2f6a232baef1e0c P eeee1578317046a7acff34fd57ab3022 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:11.049288 12669 raft_consensus.cc:1261] T 1a887236b132491db2f6a232baef1e0c P 38da097b32ae4db9a2d2ad6f5e03d266 [term 1 FOLLOWER]: Refusing update from remote peer 5b244d023c7247219f0d39a360b3e3e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:11.049901 13067 consensus_queue.cc:1035] T 1a887236b132491db2f6a232baef1e0c P 5b244d023c7247219f0d39a360b3e3e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:11.050347 12805 raft_consensus.cc:1261] T 1a887236b132491db2f6a232baef1e0c P eeee1578317046a7acff34fd57ab3022 [term 1 FOLLOWER]: Refusing update from remote peer 5b244d023c7247219f0d39a360b3e3e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:11.051010 13067 consensus_queue.cc:1035] T 1a887236b132491db2f6a232baef1e0c P 5b244d023c7247219f0d39a360b3e3e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:11.082086 13069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:11.076101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:39699 (local address 127.10.131.3:35431)
0620 18:32:11.076270 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:11.076277 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:11.076435 (+ 158us) server_negotiation.cc:407] Connection header received
0620 18:32:11.076681 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:11.076686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:11.076766 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:11.076919 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:32:11.077337 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:11.078467 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:11.079931 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:11.080596 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:11.080754 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:11.081274 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:32:11.081412 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:32:11.081512 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:11.081669 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 18:32:11.081748 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:32:11.082124 13052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:11.076008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:35431 (local address 127.10.131.2:39699)
0620 18:32:11.076329 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:11.076347 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:11.076513 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:11.076938 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:11.076943 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:11.076954 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:32:11.077167 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:11.077182 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:11.078589 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:11.078593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:11.079765 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:11.079779 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:11.080800 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:11.080810 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:11.081044 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:11.081090 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:32:11.081524 (+ 434us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:32:11.081538 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 18:32:11.081631 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 18:32:11.081802 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184}
W20260620 18:32:11.093463 13042 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 18:32:12.708472 12509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:32:12.708699 12509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:32:12.712749 12509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f77c29104bae4b95b9f3c560d0589bb9: Sending DeleteTablet for 3 replicas of tablet 1a887236b132491db2f6a232baef1e0c
I20260620 18:32:12.713539 12646 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a887236b132491db2f6a232baef1e0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:12 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:60390
I20260620 18:32:12.713606 12926 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a887236b132491db2f6a232baef1e0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:12 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:59370
I20260620 18:32:12.713651 12786 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a887236b132491db2f6a232baef1e0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:12 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:50920
I20260620 18:32:12.725725 13101 tablet_replica.cc:330] T 1a887236b132491db2f6a232baef1e0c P 38da097b32ae4db9a2d2ad6f5e03d266: stopping tablet replica
I20260620 18:32:12.725968 13101 raft_consensus.cc:2229] T 1a887236b132491db2f6a232baef1e0c P 38da097b32ae4db9a2d2ad6f5e03d266 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:32:12.726087 13102 tablet_replica.cc:330] T 1a887236b132491db2f6a232baef1e0c P 5b244d023c7247219f0d39a360b3e3e0: stopping tablet replica
I20260620 18:32:12.726236 13101 raft_consensus.cc:2258] T 1a887236b132491db2f6a232baef1e0c P 38da097b32ae4db9a2d2ad6f5e03d266 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:12.726451 13102 raft_consensus.cc:2229] T 1a887236b132491db2f6a232baef1e0c P 5b244d023c7247219f0d39a360b3e3e0 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:32:12.727142 13102 raft_consensus.cc:2258] T 1a887236b132491db2f6a232baef1e0c P 5b244d023c7247219f0d39a360b3e3e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:12.727401 13101 ts_tablet_manager.cc:1905] T 1a887236b132491db2f6a232baef1e0c P 38da097b32ae4db9a2d2ad6f5e03d266: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:12.728608 13102 ts_tablet_manager.cc:1905] T 1a887236b132491db2f6a232baef1e0c P 5b244d023c7247219f0d39a360b3e3e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:12.729837 13103 tablet_replica.cc:330] T 1a887236b132491db2f6a232baef1e0c P eeee1578317046a7acff34fd57ab3022: stopping tablet replica
I20260620 18:32:12.730069 13103 raft_consensus.cc:2229] T 1a887236b132491db2f6a232baef1e0c P eeee1578317046a7acff34fd57ab3022 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:32:12.730309 13103 raft_consensus.cc:2258] T 1a887236b132491db2f6a232baef1e0c P eeee1578317046a7acff34fd57ab3022 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:12.731205 13103 ts_tablet_manager.cc:1905] T 1a887236b132491db2f6a232baef1e0c P eeee1578317046a7acff34fd57ab3022: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:12.735088 13101 ts_tablet_manager.cc:1918] T 1a887236b132491db2f6a232baef1e0c P 38da097b32ae4db9a2d2ad6f5e03d266: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:32:12.735234 13101 log.cc:1196] T 1a887236b132491db2f6a232baef1e0c P 38da097b32ae4db9a2d2ad6f5e03d266: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-0/wal/wals/1a887236b132491db2f6a232baef1e0c
I20260620 18:32:12.735672 13101 ts_tablet_manager.cc:1939] T 1a887236b132491db2f6a232baef1e0c P 38da097b32ae4db9a2d2ad6f5e03d266: Deleting consensus metadata
I20260620 18:32:12.736699 13102 ts_tablet_manager.cc:1918] T 1a887236b132491db2f6a232baef1e0c P 5b244d023c7247219f0d39a360b3e3e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:32:12.736796 12493 catalog_manager.cc:4841] TS 38da097b32ae4db9a2d2ad6f5e03d266 (127.10.131.1:41907): tablet 1a887236b132491db2f6a232baef1e0c (table test-table [id=48fd067c74c54e11b862ca5984f8a564]) successfully deleted
I20260620 18:32:12.736922 13102 log.cc:1196] T 1a887236b132491db2f6a232baef1e0c P 5b244d023c7247219f0d39a360b3e3e0: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-2/wal/wals/1a887236b132491db2f6a232baef1e0c
I20260620 18:32:12.737473 13102 ts_tablet_manager.cc:1939] T 1a887236b132491db2f6a232baef1e0c P 5b244d023c7247219f0d39a360b3e3e0: Deleting consensus metadata
I20260620 18:32:12.737855 13103 ts_tablet_manager.cc:1918] T 1a887236b132491db2f6a232baef1e0c P eeee1578317046a7acff34fd57ab3022: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:32:12.737991 13103 log.cc:1196] T 1a887236b132491db2f6a232baef1e0c P eeee1578317046a7acff34fd57ab3022: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-1/wal/wals/1a887236b132491db2f6a232baef1e0c
I20260620 18:32:12.738430 13103 ts_tablet_manager.cc:1939] T 1a887236b132491db2f6a232baef1e0c P eeee1578317046a7acff34fd57ab3022: Deleting consensus metadata
I20260620 18:32:12.738608 12494 catalog_manager.cc:4841] TS 5b244d023c7247219f0d39a360b3e3e0 (127.10.131.3:35431): tablet 1a887236b132491db2f6a232baef1e0c (table test-table [id=48fd067c74c54e11b862ca5984f8a564]) successfully deleted
I20260620 18:32:12.739339 12494 catalog_manager.cc:4841] TS eeee1578317046a7acff34fd57ab3022 (127.10.131.2:36507): tablet 1a887236b132491db2f6a232baef1e0c (table test-table [id=48fd067c74c54e11b862ca5984f8a564]) successfully deleted
Jun 20 18:32:12 dist-test-slave-cbn3 krb5kdc[12464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980332, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 18:32:12 dist-test-slave-cbn3 krb5kdc[12464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980332, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:32:12.773545 13111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:12.759309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52518 (local address 127.10.131.62:43781)
0620 18:32:12.759985 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:12.759990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:12.760016 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:32:12.760220 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:12.760224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:12.760289 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:12.760386 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:12.761871 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:12.762721 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:12.764193 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:12.764707 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:12.764822 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:12.768637 (+ 3815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:12.768643 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:12.768646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:12.768680 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:12.771106 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:12.771689 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:12.771692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:12.771694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:12.771805 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:12.772134 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:12.772138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:12.772140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:12.772449 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:12.772628 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:12.772941 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 18:32:12.773123 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":331,"threads_started":1}
I20260620 18:32:12.777201 12509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:32:12.777789 12509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:32:12.783932 12646 tablet_service.cc:1467] Processing CreateTablet for tablet 4b7ce4bacd334352bda91ef060e2f087 (DEFAULT_TABLE table=test-table [id=258204e315a64666ab9e2551607122ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:12.784148 12786 tablet_service.cc:1467] Processing CreateTablet for tablet 4b7ce4bacd334352bda91ef060e2f087 (DEFAULT_TABLE table=test-table [id=258204e315a64666ab9e2551607122ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:12.784217 12926 tablet_service.cc:1467] Processing CreateTablet for tablet 4b7ce4bacd334352bda91ef060e2f087 (DEFAULT_TABLE table=test-table [id=258204e315a64666ab9e2551607122ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:12.784368 12646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b7ce4bacd334352bda91ef060e2f087. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:12.784547 12786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b7ce4bacd334352bda91ef060e2f087. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:12.784592 12926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b7ce4bacd334352bda91ef060e2f087. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:12.790484 13112 tablet_bootstrap.cc:492] T 4b7ce4bacd334352bda91ef060e2f087 P eeee1578317046a7acff34fd57ab3022: Bootstrap starting.
I20260620 18:32:12.791121 13114 tablet_bootstrap.cc:492] T 4b7ce4bacd334352bda91ef060e2f087 P 38da097b32ae4db9a2d2ad6f5e03d266: Bootstrap starting.
I20260620 18:32:12.792537 13112 tablet_bootstrap.cc:654] T 4b7ce4bacd334352bda91ef060e2f087 P eeee1578317046a7acff34fd57ab3022: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:12.792768 13114 tablet_bootstrap.cc:654] T 4b7ce4bacd334352bda91ef060e2f087 P 38da097b32ae4db9a2d2ad6f5e03d266: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:12.794180 13113 tablet_bootstrap.cc:492] T 4b7ce4bacd334352bda91ef060e2f087 P 5b244d023c7247219f0d39a360b3e3e0: Bootstrap starting.
I20260620 18:32:12.795193 13112 tablet_bootstrap.cc:492] T 4b7ce4bacd334352bda91ef060e2f087 P eeee1578317046a7acff34fd57ab3022: No bootstrap required, opened a new log
I20260620 18:32:12.795332 13112 ts_tablet_manager.cc:1397] T 4b7ce4bacd334352bda91ef060e2f087 P eeee1578317046a7acff34fd57ab3022: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 18:32:12.795549 13114 tablet_bootstrap.cc:492] T 4b7ce4bacd334352bda91ef060e2f087 P 38da097b32ae4db9a2d2ad6f5e03d266: No bootstrap required, opened a new log
I20260620 18:32:12.795679 13114 ts_tablet_manager.cc:1397] T 4b7ce4bacd334352bda91ef060e2f087 P 38da097b32ae4db9a2d2ad6f5e03d266: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 18:32:12.795799 13113 tablet_bootstrap.cc:654] T 4b7ce4bacd334352bda91ef060e2f087 P 5b244d023c7247219f0d39a360b3e3e0: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:12.796178 13112 raft_consensus.cc:348] T 4b7ce4bacd334352bda91ef060e2f087 P eeee1578317046a7acff34fd57ab3022 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } }
I20260620 18:32:12.796447 13112 raft_consensus.cc:374] T 4b7ce4bacd334352bda91ef060e2f087 P eeee1578317046a7acff34fd57ab3022 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:12.796387 13114 raft_consensus.cc:348] T 4b7ce4bacd334352bda91ef060e2f087 P 38da097b32ae4db9a2d2ad6f5e03d266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } }
I20260620 18:32:12.796541 13112 raft_consensus.cc:729] T 4b7ce4bacd334352bda91ef060e2f087 P eeee1578317046a7acff34fd57ab3022 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeee1578317046a7acff34fd57ab3022, State: Initialized, Role: FOLLOWER
I20260620 18:32:12.796630 13114 raft_consensus.cc:374] T 4b7ce4bacd334352bda91ef060e2f087 P 38da097b32ae4db9a2d2ad6f5e03d266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:12.796782 13114 raft_consensus.cc:729] T 4b7ce4bacd334352bda91ef060e2f087 P 38da097b32ae4db9a2d2ad6f5e03d266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38da097b32ae4db9a2d2ad6f5e03d266, State: Initialized, Role: FOLLOWER
I20260620 18:32:12.797029 13114 consensus_queue.cc:260] T 4b7ce4bacd334352bda91ef060e2f087 P 38da097b32ae4db9a2d2ad6f5e03d266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } }
I20260620 18:32:12.796782 13112 consensus_queue.cc:260] T 4b7ce4bacd334352bda91ef060e2f087 P eeee1578317046a7acff34fd57ab3022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } }
I20260620 18:32:12.797636 13112 ts_tablet_manager.cc:1428] T 4b7ce4bacd334352bda91ef060e2f087 P eeee1578317046a7acff34fd57ab3022: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:32:12.798152 13113 tablet_bootstrap.cc:492] T 4b7ce4bacd334352bda91ef060e2f087 P 5b244d023c7247219f0d39a360b3e3e0: No bootstrap required, opened a new log
I20260620 18:32:12.798339 13113 ts_tablet_manager.cc:1397] T 4b7ce4bacd334352bda91ef060e2f087 P 5b244d023c7247219f0d39a360b3e3e0: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 18:32:12.798997 13114 ts_tablet_manager.cc:1428] T 4b7ce4bacd334352bda91ef060e2f087 P 38da097b32ae4db9a2d2ad6f5e03d266: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 18:32:12.799002 13113 raft_consensus.cc:348] T 4b7ce4bacd334352bda91ef060e2f087 P 5b244d023c7247219f0d39a360b3e3e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } }
I20260620 18:32:12.799245 13113 raft_consensus.cc:374] T 4b7ce4bacd334352bda91ef060e2f087 P 5b244d023c7247219f0d39a360b3e3e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:12.799345 13113 raft_consensus.cc:729] T 4b7ce4bacd334352bda91ef060e2f087 P 5b244d023c7247219f0d39a360b3e3e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b244d023c7247219f0d39a360b3e3e0, State: Initialized, Role: FOLLOWER
I20260620 18:32:12.799590 13113 consensus_queue.cc:260] T 4b7ce4bacd334352bda91ef060e2f087 P 5b244d023c7247219f0d39a360b3e3e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } }
I20260620 18:32:12.800207 13113 ts_tablet_manager.cc:1428] T 4b7ce4bacd334352bda91ef060e2f087 P 5b244d023c7247219f0d39a360b3e3e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:32:12.930881 13118 raft_consensus.cc:482] T 4b7ce4bacd334352bda91ef060e2f087 P 38da097b32ae4db9a2d2ad6f5e03d266 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:32:12.931133 13118 raft_consensus.cc:504] T 4b7ce4bacd334352bda91ef060e2f087 P 38da097b32ae4db9a2d2ad6f5e03d266 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } }
I20260620 18:32:12.932672 13118 leader_election.cc:288] T 4b7ce4bacd334352bda91ef060e2f087 P 38da097b32ae4db9a2d2ad6f5e03d266 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eeee1578317046a7acff34fd57ab3022 (127.10.131.2:36507), 5b244d023c7247219f0d39a360b3e3e0 (127.10.131.3:35431)
I20260620 18:32:12.940043 13120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:12.933106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:43247 (local address 127.10.131.2:36507)
0620 18:32:12.933829 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:12.933835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:12.933863 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:32:12.934124 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:12.934132 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:12.934225 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:12.934327 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:12.935064 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:12.936102 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:12.938321 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:12.939070 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:12.939250 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:12.939326 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:12.939580 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 18:32:12.939710 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":384,"threads_started":1}
I20260620 18:32:12.940079 13119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:12.932931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:36507 (local address 127.10.131.1:43247)
0620 18:32:12.933693 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:12.933723 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:12.933890 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:12.934396 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:12.934402 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:12.934430 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:12.934814 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:12.934827 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:12.936254 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:12.936260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:12.938178 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:12.938189 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:12.939257 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:12.939263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:12.939422 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:12.939445 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 18:32:12.939554 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 18:32:12.939680 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":1,"thread_start_us":382,"threads_started":1}
I20260620 18:32:12.940528 13122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:12.934181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:57123 (local address 127.10.131.3:35431)
0620 18:32:12.934922 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:12.934929 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:12.934950 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:32:12.935032 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:12.935035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:12.935086 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:12.935156 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:12.935744 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:12.936752 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:12.938796 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:12.939551 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:12.939729 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:12.939800 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:12.940071 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 18:32:12.940223 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":438,"threads_started":1}
I20260620 18:32:12.940622 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b7ce4bacd334352bda91ef060e2f087" candidate_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeee1578317046a7acff34fd57ab3022" is_pre_election: true
I20260620 18:32:12.940958 13121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:12.933779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:35431 (local address 127.10.131.1:57123)
0620 18:32:12.934600 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:12.934625 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:12.934775 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:12.935220 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:12.935225 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:12.935252 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:12.935591 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:12.935603 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:12.936912 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:12.936920 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:12.938645 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:12.938656 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:12.939750 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:12.939757 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:12.939912 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:12.939935 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 18:32:12.940047 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 18:32:12.940496 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"mutex_wait_us":50,"thread_start_us":402,"threads_started":1}
I20260620 18:32:12.940920 12805 raft_consensus.cc:2454] T 4b7ce4bacd334352bda91ef060e2f087 P eeee1578317046a7acff34fd57ab3022 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38da097b32ae4db9a2d2ad6f5e03d266 in term 0.
I20260620 18:32:12.941408 12597 leader_election.cc:302] T 4b7ce4bacd334352bda91ef060e2f087 P 38da097b32ae4db9a2d2ad6f5e03d266 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38da097b32ae4db9a2d2ad6f5e03d266, eeee1578317046a7acff34fd57ab3022; no voters:
I20260620 18:32:12.941598 12951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b7ce4bacd334352bda91ef060e2f087" candidate_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b244d023c7247219f0d39a360b3e3e0" is_pre_election: true
I20260620 18:32:12.941828 13118 raft_consensus.cc:2781] T 4b7ce4bacd334352bda91ef060e2f087 P 38da097b32ae4db9a2d2ad6f5e03d266 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:32:12.941824 12951 raft_consensus.cc:2454] T 4b7ce4bacd334352bda91ef060e2f087 P 5b244d023c7247219f0d39a360b3e3e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38da097b32ae4db9a2d2ad6f5e03d266 in term 0.
I20260620 18:32:12.941960 13118 raft_consensus.cc:482] T 4b7ce4bacd334352bda91ef060e2f087 P 38da097b32ae4db9a2d2ad6f5e03d266 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:32:12.942070 13118 raft_consensus.cc:3037] T 4b7ce4bacd334352bda91ef060e2f087 P 38da097b32ae4db9a2d2ad6f5e03d266 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:12.944134 13118 raft_consensus.cc:504] T 4b7ce4bacd334352bda91ef060e2f087 P 38da097b32ae4db9a2d2ad6f5e03d266 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } }
I20260620 18:32:12.944682 13118 leader_election.cc:288] T 4b7ce4bacd334352bda91ef060e2f087 P 38da097b32ae4db9a2d2ad6f5e03d266 [CANDIDATE]: Term 1 election: Requested vote from peers eeee1578317046a7acff34fd57ab3022 (127.10.131.2:36507), 5b244d023c7247219f0d39a360b3e3e0 (127.10.131.3:35431)
I20260620 18:32:12.945164 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b7ce4bacd334352bda91ef060e2f087" candidate_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeee1578317046a7acff34fd57ab3022"
I20260620 18:32:12.945223 12951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b7ce4bacd334352bda91ef060e2f087" candidate_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b244d023c7247219f0d39a360b3e3e0"
I20260620 18:32:12.945377 12805 raft_consensus.cc:3037] T 4b7ce4bacd334352bda91ef060e2f087 P eeee1578317046a7acff34fd57ab3022 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:12.945408 12951 raft_consensus.cc:3037] T 4b7ce4bacd334352bda91ef060e2f087 P 5b244d023c7247219f0d39a360b3e3e0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:12.947305 12951 raft_consensus.cc:2454] T 4b7ce4bacd334352bda91ef060e2f087 P 5b244d023c7247219f0d39a360b3e3e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38da097b32ae4db9a2d2ad6f5e03d266 in term 1.
I20260620 18:32:12.947360 12805 raft_consensus.cc:2454] T 4b7ce4bacd334352bda91ef060e2f087 P eeee1578317046a7acff34fd57ab3022 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38da097b32ae4db9a2d2ad6f5e03d266 in term 1.
I20260620 18:32:12.947659 12597 leader_election.cc:302] T 4b7ce4bacd334352bda91ef060e2f087 P 38da097b32ae4db9a2d2ad6f5e03d266 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38da097b32ae4db9a2d2ad6f5e03d266, 5b244d023c7247219f0d39a360b3e3e0; no voters:
I20260620 18:32:12.947947 13118 raft_consensus.cc:2781] T 4b7ce4bacd334352bda91ef060e2f087 P 38da097b32ae4db9a2d2ad6f5e03d266 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:12.948354 13118 raft_consensus.cc:686] T 4b7ce4bacd334352bda91ef060e2f087 P 38da097b32ae4db9a2d2ad6f5e03d266 [term 1 LEADER]: Becoming Leader. State: Replica: 38da097b32ae4db9a2d2ad6f5e03d266, State: Running, Role: LEADER
I20260620 18:32:12.948750 13118 consensus_queue.cc:237] T 4b7ce4bacd334352bda91ef060e2f087 P 38da097b32ae4db9a2d2ad6f5e03d266 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } }
I20260620 18:32:12.951820 12509 catalog_manager.cc:5495] T 4b7ce4bacd334352bda91ef060e2f087 P 38da097b32ae4db9a2d2ad6f5e03d266 reported cstate change: term changed from 0 to 1, leader changed from <none> to 38da097b32ae4db9a2d2ad6f5e03d266 (127.10.131.1). New cstate: current_term: 1 leader_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } health_report { overall_health: UNKNOWN } } }
I20260620 18:32:12.992722 13126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:12.987173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60396 (local address 127.10.131.1:41907)
0620 18:32:12.987721 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:12.987725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:12.987743 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:32:12.987818 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:12.987821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:12.987889 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:12.988055 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:32:12.988464 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:12.989266 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:12.990569 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:12.990970 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:12.991102 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:12.991550 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:32:12.992101 (+ 551us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:32:12.992259 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:12.992315 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 18:32:12.992388 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":279,"threads_started":1}
I20260620 18:32:13.006843 13120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:13.001600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:49963 (local address 127.10.131.2:36507)
0620 18:32:13.001813 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:13.001821 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:13.002169 (+ 348us) server_negotiation.cc:407] Connection header received
0620 18:32:13.002368 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:13.002372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:13.002430 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:13.002545 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:32:13.002995 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:13.003655 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:13.005214 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:13.005598 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:13.005737 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:13.006138 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:32:13.006238 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:32:13.006349 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:13.006489 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 18:32:13.006563 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:32:13.006943 13129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:13.001382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:36507 (local address 127.10.131.1:49963)
0620 18:32:13.002075 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:13.002090 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:13.002240 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:13.002570 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:13.002574 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:13.002586 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:32:13.002859 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:13.002867 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:13.003787 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:13.003792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:13.005065 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:13.005076 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:13.005764 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:13.005770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:13.005914 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:13.005962 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:32:13.006350 (+ 388us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:32:13.006364 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 18:32:13.006461 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 18:32:13.006585 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":362,"threads_started":1}
I20260620 18:32:13.015018 12805 raft_consensus.cc:1261] T 4b7ce4bacd334352bda91ef060e2f087 P eeee1578317046a7acff34fd57ab3022 [term 1 FOLLOWER]: Refusing update from remote peer 38da097b32ae4db9a2d2ad6f5e03d266: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:13.015084 12951 raft_consensus.cc:1261] T 4b7ce4bacd334352bda91ef060e2f087 P 5b244d023c7247219f0d39a360b3e3e0 [term 1 FOLLOWER]: Refusing update from remote peer 38da097b32ae4db9a2d2ad6f5e03d266: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:13.015633 13123 consensus_queue.cc:1035] T 4b7ce4bacd334352bda91ef060e2f087 P 38da097b32ae4db9a2d2ad6f5e03d266 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:13.015982 13118 consensus_queue.cc:1035] T 4b7ce4bacd334352bda91ef060e2f087 P 38da097b32ae4db9a2d2ad6f5e03d266 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:13.042271 13126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:13.035389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:51221 (local address 127.10.131.1:41907)
0620 18:32:13.035616 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:13.035624 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:13.037425 (+ 1801us) server_negotiation.cc:407] Connection header received
0620 18:32:13.037620 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:13.037635 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:13.037716 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:13.037811 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:32:13.038337 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:13.039248 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:13.040511 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:13.041020 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:13.041135 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:13.041517 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:32:13.041678 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:32:13.041768 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:13.041911 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 18:32:13.041969 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:32:13.042358 13134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:13.035313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:41907 (local address 127.10.131.2:51221)
0620 18:32:13.037321 (+ 2008us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:13.037341 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:13.037479 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:13.037853 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:13.037857 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:13.037869 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:32:13.038181 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:13.038196 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:13.039367 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:13.039371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:13.040399 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:13.040410 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:13.041181 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:13.041190 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:13.041319 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:13.041353 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:32:13.041782 (+ 429us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:32:13.041795 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 18:32:13.041887 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 18:32:13.042013 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1846,"thread_start_us":1734,"threads_started":1}
I20260620 18:32:13.070370 12508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:32:13.070636 12508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:32:13.073750 12508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f77c29104bae4b95b9f3c560d0589bb9: Sending DeleteTablet for 3 replicas of tablet 4b7ce4bacd334352bda91ef060e2f087
I20260620 18:32:13.074467 12646 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b7ce4bacd334352bda91ef060e2f087 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:13 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:60390
I20260620 18:32:13.074787 13101 tablet_replica.cc:330] T 4b7ce4bacd334352bda91ef060e2f087 P 38da097b32ae4db9a2d2ad6f5e03d266: stopping tablet replica
I20260620 18:32:13.074836 12786 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b7ce4bacd334352bda91ef060e2f087 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:13 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:50920
I20260620 18:32:13.074973 13101 raft_consensus.cc:2229] T 4b7ce4bacd334352bda91ef060e2f087 P 38da097b32ae4db9a2d2ad6f5e03d266 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:32:13.075145 12926 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b7ce4bacd334352bda91ef060e2f087 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:13 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:59370
I20260620 18:32:13.075418 13103 tablet_replica.cc:330] T 4b7ce4bacd334352bda91ef060e2f087 P eeee1578317046a7acff34fd57ab3022: stopping tablet replica
I20260620 18:32:13.075446 13101 raft_consensus.cc:2258] T 4b7ce4bacd334352bda91ef060e2f087 P 38da097b32ae4db9a2d2ad6f5e03d266 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:13.075620 13103 raft_consensus.cc:2229] T 4b7ce4bacd334352bda91ef060e2f087 P eeee1578317046a7acff34fd57ab3022 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:32:13.075496 12508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 18:32:13.075711 13102 tablet_replica.cc:330] T 4b7ce4bacd334352bda91ef060e2f087 P 5b244d023c7247219f0d39a360b3e3e0: stopping tablet replica
I20260620 18:32:13.075803 13103 raft_consensus.cc:2258] T 4b7ce4bacd334352bda91ef060e2f087 P eeee1578317046a7acff34fd57ab3022 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:13.075918 13102 raft_consensus.cc:2229] T 4b7ce4bacd334352bda91ef060e2f087 P 5b244d023c7247219f0d39a360b3e3e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:32:13.076107 13102 raft_consensus.cc:2258] T 4b7ce4bacd334352bda91ef060e2f087 P 5b244d023c7247219f0d39a360b3e3e0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 18:32:13.076161 12508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:32:13.077260 13103 ts_tablet_manager.cc:1905] T 4b7ce4bacd334352bda91ef060e2f087 P eeee1578317046a7acff34fd57ab3022: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:13.077456 13102 ts_tablet_manager.cc:1905] T 4b7ce4bacd334352bda91ef060e2f087 P 5b244d023c7247219f0d39a360b3e3e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:13.080821 13101 ts_tablet_manager.cc:1905] T 4b7ce4bacd334352bda91ef060e2f087 P 38da097b32ae4db9a2d2ad6f5e03d266: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:13.083823 13102 ts_tablet_manager.cc:1918] T 4b7ce4bacd334352bda91ef060e2f087 P 5b244d023c7247219f0d39a360b3e3e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:32:13.083823 13103 ts_tablet_manager.cc:1918] T 4b7ce4bacd334352bda91ef060e2f087 P eeee1578317046a7acff34fd57ab3022: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:32:13.083958 13102 log.cc:1196] T 4b7ce4bacd334352bda91ef060e2f087 P 5b244d023c7247219f0d39a360b3e3e0: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-2/wal/wals/4b7ce4bacd334352bda91ef060e2f087
I20260620 18:32:13.083958 13103 log.cc:1196] T 4b7ce4bacd334352bda91ef060e2f087 P eeee1578317046a7acff34fd57ab3022: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-1/wal/wals/4b7ce4bacd334352bda91ef060e2f087
I20260620 18:32:13.084363 13103 ts_tablet_manager.cc:1939] T 4b7ce4bacd334352bda91ef060e2f087 P eeee1578317046a7acff34fd57ab3022: Deleting consensus metadata
I20260620 18:32:13.084363 13102 ts_tablet_manager.cc:1939] T 4b7ce4bacd334352bda91ef060e2f087 P 5b244d023c7247219f0d39a360b3e3e0: Deleting consensus metadata
I20260620 18:32:13.085394 12494 catalog_manager.cc:4841] TS eeee1578317046a7acff34fd57ab3022 (127.10.131.2:36507): tablet 4b7ce4bacd334352bda91ef060e2f087 (table test-table [id=258204e315a64666ab9e2551607122ca]) successfully deleted
I20260620 18:32:13.085657 12494 catalog_manager.cc:4841] TS 5b244d023c7247219f0d39a360b3e3e0 (127.10.131.3:35431): tablet 4b7ce4bacd334352bda91ef060e2f087 (table test-table [id=258204e315a64666ab9e2551607122ca]) successfully deleted
I20260620 18:32:13.089390 13101 ts_tablet_manager.cc:1918] T 4b7ce4bacd334352bda91ef060e2f087 P 38da097b32ae4db9a2d2ad6f5e03d266: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:32:13.089562 13101 log.cc:1196] T 4b7ce4bacd334352bda91ef060e2f087 P 38da097b32ae4db9a2d2ad6f5e03d266: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-0/wal/wals/4b7ce4bacd334352bda91ef060e2f087
I20260620 18:32:13.089985 13101 ts_tablet_manager.cc:1939] T 4b7ce4bacd334352bda91ef060e2f087 P 38da097b32ae4db9a2d2ad6f5e03d266: Deleting consensus metadata
I20260620 18:32:13.091310 12493 catalog_manager.cc:4841] TS 38da097b32ae4db9a2d2ad6f5e03d266 (127.10.131.1:41907): tablet 4b7ce4bacd334352bda91ef060e2f087 (table test-table [id=258204e315a64666ab9e2551607122ca]) successfully deleted
I20260620 18:32:13.092140 12646 tablet_service.cc:1467] Processing CreateTablet for tablet cb4b1178fe1e4a2ba342c24c26990a84 (DEFAULT_TABLE table=test-table [id=85606a0651664c65a87a632a3f0acb72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:13.092188 12786 tablet_service.cc:1467] Processing CreateTablet for tablet cb4b1178fe1e4a2ba342c24c26990a84 (DEFAULT_TABLE table=test-table [id=85606a0651664c65a87a632a3f0acb72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:13.092535 12786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb4b1178fe1e4a2ba342c24c26990a84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:13.092554 12646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb4b1178fe1e4a2ba342c24c26990a84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:13.097705 12926 tablet_service.cc:1467] Processing CreateTablet for tablet cb4b1178fe1e4a2ba342c24c26990a84 (DEFAULT_TABLE table=test-table [id=85606a0651664c65a87a632a3f0acb72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:13.098160 12926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb4b1178fe1e4a2ba342c24c26990a84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:13.099049 13112 tablet_bootstrap.cc:492] T cb4b1178fe1e4a2ba342c24c26990a84 P eeee1578317046a7acff34fd57ab3022: Bootstrap starting.
I20260620 18:32:13.100668 13112 tablet_bootstrap.cc:654] T cb4b1178fe1e4a2ba342c24c26990a84 P eeee1578317046a7acff34fd57ab3022: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:13.100890 13114 tablet_bootstrap.cc:492] T cb4b1178fe1e4a2ba342c24c26990a84 P 38da097b32ae4db9a2d2ad6f5e03d266: Bootstrap starting.
I20260620 18:32:13.103402 13113 tablet_bootstrap.cc:492] T cb4b1178fe1e4a2ba342c24c26990a84 P 5b244d023c7247219f0d39a360b3e3e0: Bootstrap starting.
I20260620 18:32:13.104907 13112 tablet_bootstrap.cc:492] T cb4b1178fe1e4a2ba342c24c26990a84 P eeee1578317046a7acff34fd57ab3022: No bootstrap required, opened a new log
I20260620 18:32:13.104984 13114 tablet_bootstrap.cc:654] T cb4b1178fe1e4a2ba342c24c26990a84 P 38da097b32ae4db9a2d2ad6f5e03d266: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:13.105090 13112 ts_tablet_manager.cc:1397] T cb4b1178fe1e4a2ba342c24c26990a84 P eeee1578317046a7acff34fd57ab3022: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260620 18:32:13.105140 13113 tablet_bootstrap.cc:654] T cb4b1178fe1e4a2ba342c24c26990a84 P 5b244d023c7247219f0d39a360b3e3e0: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:13.105883 13112 raft_consensus.cc:348] T cb4b1178fe1e4a2ba342c24c26990a84 P eeee1578317046a7acff34fd57ab3022 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } }
I20260620 18:32:13.106189 13112 raft_consensus.cc:374] T cb4b1178fe1e4a2ba342c24c26990a84 P eeee1578317046a7acff34fd57ab3022 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:13.106273 13112 raft_consensus.cc:729] T cb4b1178fe1e4a2ba342c24c26990a84 P eeee1578317046a7acff34fd57ab3022 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeee1578317046a7acff34fd57ab3022, State: Initialized, Role: FOLLOWER
I20260620 18:32:13.106559 13112 consensus_queue.cc:260] T cb4b1178fe1e4a2ba342c24c26990a84 P eeee1578317046a7acff34fd57ab3022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } }
I20260620 18:32:13.108080 13112 ts_tablet_manager.cc:1428] T cb4b1178fe1e4a2ba342c24c26990a84 P eeee1578317046a7acff34fd57ab3022: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 18:32:13.109824 13114 tablet_bootstrap.cc:492] T cb4b1178fe1e4a2ba342c24c26990a84 P 38da097b32ae4db9a2d2ad6f5e03d266: No bootstrap required, opened a new log
I20260620 18:32:13.110009 13114 ts_tablet_manager.cc:1397] T cb4b1178fe1e4a2ba342c24c26990a84 P 38da097b32ae4db9a2d2ad6f5e03d266: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20260620 18:32:13.110818 13114 raft_consensus.cc:348] T cb4b1178fe1e4a2ba342c24c26990a84 P 38da097b32ae4db9a2d2ad6f5e03d266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } }
I20260620 18:32:13.111090 13114 raft_consensus.cc:374] T cb4b1178fe1e4a2ba342c24c26990a84 P 38da097b32ae4db9a2d2ad6f5e03d266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:13.111171 13114 raft_consensus.cc:729] T cb4b1178fe1e4a2ba342c24c26990a84 P 38da097b32ae4db9a2d2ad6f5e03d266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38da097b32ae4db9a2d2ad6f5e03d266, State: Initialized, Role: FOLLOWER
I20260620 18:32:13.111490 13114 consensus_queue.cc:260] T cb4b1178fe1e4a2ba342c24c26990a84 P 38da097b32ae4db9a2d2ad6f5e03d266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } }
I20260620 18:32:13.112048 13114 ts_tablet_manager.cc:1428] T cb4b1178fe1e4a2ba342c24c26990a84 P 38da097b32ae4db9a2d2ad6f5e03d266: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 18:32:13.117924 13113 tablet_bootstrap.cc:492] T cb4b1178fe1e4a2ba342c24c26990a84 P 5b244d023c7247219f0d39a360b3e3e0: No bootstrap required, opened a new log
I20260620 18:32:13.118130 13113 ts_tablet_manager.cc:1397] T cb4b1178fe1e4a2ba342c24c26990a84 P 5b244d023c7247219f0d39a360b3e3e0: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.008s
I20260620 18:32:13.118829 13113 raft_consensus.cc:348] T cb4b1178fe1e4a2ba342c24c26990a84 P 5b244d023c7247219f0d39a360b3e3e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } }
I20260620 18:32:13.119058 13113 raft_consensus.cc:374] T cb4b1178fe1e4a2ba342c24c26990a84 P 5b244d023c7247219f0d39a360b3e3e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:13.119160 13113 raft_consensus.cc:729] T cb4b1178fe1e4a2ba342c24c26990a84 P 5b244d023c7247219f0d39a360b3e3e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b244d023c7247219f0d39a360b3e3e0, State: Initialized, Role: FOLLOWER
I20260620 18:32:13.119388 13113 consensus_queue.cc:260] T cb4b1178fe1e4a2ba342c24c26990a84 P 5b244d023c7247219f0d39a360b3e3e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } }
I20260620 18:32:13.119894 13113 ts_tablet_manager.cc:1428] T cb4b1178fe1e4a2ba342c24c26990a84 P 5b244d023c7247219f0d39a360b3e3e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 18:32:13.173134 13053 raft_consensus.cc:482] T cb4b1178fe1e4a2ba342c24c26990a84 P eeee1578317046a7acff34fd57ab3022 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:32:13.173388 13053 raft_consensus.cc:504] T cb4b1178fe1e4a2ba342c24c26990a84 P eeee1578317046a7acff34fd57ab3022 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } }
I20260620 18:32:13.174113 13053 leader_election.cc:288] T cb4b1178fe1e4a2ba342c24c26990a84 P eeee1578317046a7acff34fd57ab3022 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38da097b32ae4db9a2d2ad6f5e03d266 (127.10.131.1:41907), 5b244d023c7247219f0d39a360b3e3e0 (127.10.131.3:35431)
I20260620 18:32:13.174633 12669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4b1178fe1e4a2ba342c24c26990a84" candidate_uuid: "eeee1578317046a7acff34fd57ab3022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" is_pre_election: true
I20260620 18:32:13.174656 12951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4b1178fe1e4a2ba342c24c26990a84" candidate_uuid: "eeee1578317046a7acff34fd57ab3022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b244d023c7247219f0d39a360b3e3e0" is_pre_election: true
I20260620 18:32:13.174926 12669 raft_consensus.cc:2454] T cb4b1178fe1e4a2ba342c24c26990a84 P 38da097b32ae4db9a2d2ad6f5e03d266 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeee1578317046a7acff34fd57ab3022 in term 0.
I20260620 18:32:13.174907 12951 raft_consensus.cc:2454] T cb4b1178fe1e4a2ba342c24c26990a84 P 5b244d023c7247219f0d39a360b3e3e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeee1578317046a7acff34fd57ab3022 in term 0.
I20260620 18:32:13.175428 12736 leader_election.cc:302] T cb4b1178fe1e4a2ba342c24c26990a84 P eeee1578317046a7acff34fd57ab3022 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38da097b32ae4db9a2d2ad6f5e03d266, eeee1578317046a7acff34fd57ab3022; no voters:
I20260620 18:32:13.175807 13053 raft_consensus.cc:2781] T cb4b1178fe1e4a2ba342c24c26990a84 P eeee1578317046a7acff34fd57ab3022 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:32:13.175916 13053 raft_consensus.cc:482] T cb4b1178fe1e4a2ba342c24c26990a84 P eeee1578317046a7acff34fd57ab3022 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:32:13.176024 13053 raft_consensus.cc:3037] T cb4b1178fe1e4a2ba342c24c26990a84 P eeee1578317046a7acff34fd57ab3022 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:13.178225 13053 raft_consensus.cc:504] T cb4b1178fe1e4a2ba342c24c26990a84 P eeee1578317046a7acff34fd57ab3022 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } }
I20260620 18:32:13.178793 13053 leader_election.cc:288] T cb4b1178fe1e4a2ba342c24c26990a84 P eeee1578317046a7acff34fd57ab3022 [CANDIDATE]: Term 1 election: Requested vote from peers 38da097b32ae4db9a2d2ad6f5e03d266 (127.10.131.1:41907), 5b244d023c7247219f0d39a360b3e3e0 (127.10.131.3:35431)
I20260620 18:32:13.179332 12951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4b1178fe1e4a2ba342c24c26990a84" candidate_uuid: "eeee1578317046a7acff34fd57ab3022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b244d023c7247219f0d39a360b3e3e0"
I20260620 18:32:13.179247 12669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4b1178fe1e4a2ba342c24c26990a84" candidate_uuid: "eeee1578317046a7acff34fd57ab3022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38da097b32ae4db9a2d2ad6f5e03d266"
I20260620 18:32:13.179528 12951 raft_consensus.cc:3037] T cb4b1178fe1e4a2ba342c24c26990a84 P 5b244d023c7247219f0d39a360b3e3e0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:13.179545 12669 raft_consensus.cc:3037] T cb4b1178fe1e4a2ba342c24c26990a84 P 38da097b32ae4db9a2d2ad6f5e03d266 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:13.181515 12951 raft_consensus.cc:2454] T cb4b1178fe1e4a2ba342c24c26990a84 P 5b244d023c7247219f0d39a360b3e3e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeee1578317046a7acff34fd57ab3022 in term 1.
I20260620 18:32:13.181958 12737 leader_election.cc:302] T cb4b1178fe1e4a2ba342c24c26990a84 P eeee1578317046a7acff34fd57ab3022 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b244d023c7247219f0d39a360b3e3e0, eeee1578317046a7acff34fd57ab3022; no voters:
I20260620 18:32:13.182225 13053 raft_consensus.cc:2781] T cb4b1178fe1e4a2ba342c24c26990a84 P eeee1578317046a7acff34fd57ab3022 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:13.182380 13053 raft_consensus.cc:686] T cb4b1178fe1e4a2ba342c24c26990a84 P eeee1578317046a7acff34fd57ab3022 [term 1 LEADER]: Becoming Leader. State: Replica: eeee1578317046a7acff34fd57ab3022, State: Running, Role: LEADER
I20260620 18:32:13.182616 12669 raft_consensus.cc:2454] T cb4b1178fe1e4a2ba342c24c26990a84 P 38da097b32ae4db9a2d2ad6f5e03d266 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeee1578317046a7acff34fd57ab3022 in term 1.
I20260620 18:32:13.182951 13053 consensus_queue.cc:237] T cb4b1178fe1e4a2ba342c24c26990a84 P eeee1578317046a7acff34fd57ab3022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } }
I20260620 18:32:13.185767 12507 catalog_manager.cc:5495] T cb4b1178fe1e4a2ba342c24c26990a84 P eeee1578317046a7acff34fd57ab3022 reported cstate change: term changed from 0 to 1, leader changed from <none> to eeee1578317046a7acff34fd57ab3022 (127.10.131.2). New cstate: current_term: 1 leader_uuid: "eeee1578317046a7acff34fd57ab3022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeee1578317046a7acff34fd57ab3022" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 } health_report { overall_health: UNKNOWN } } }
I20260620 18:32:13.209535 13120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:13.204243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50940 (local address 127.10.131.2:36507)
0620 18:32:13.204418 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:13.204425 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:13.204640 (+ 215us) server_negotiation.cc:407] Connection header received
0620 18:32:13.204896 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:13.204900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:13.204974 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:13.205072 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:32:13.205429 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:13.206188 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:13.207672 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:13.208097 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:13.208245 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:13.208645 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:32:13.208810 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:32:13.208942 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:13.209112 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 18:32:13.209212 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 18:32:13.218674 12951 raft_consensus.cc:1261] T cb4b1178fe1e4a2ba342c24c26990a84 P 5b244d023c7247219f0d39a360b3e3e0 [term 1 FOLLOWER]: Refusing update from remote peer eeee1578317046a7acff34fd57ab3022: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:13.218698 12670 raft_consensus.cc:1261] T cb4b1178fe1e4a2ba342c24c26990a84 P 38da097b32ae4db9a2d2ad6f5e03d266 [term 1 FOLLOWER]: Refusing update from remote peer eeee1578317046a7acff34fd57ab3022: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:13.219504 13082 consensus_queue.cc:1035] T cb4b1178fe1e4a2ba342c24c26990a84 P eeee1578317046a7acff34fd57ab3022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38da097b32ae4db9a2d2ad6f5e03d266" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:13.219832 13096 consensus_queue.cc:1035] T cb4b1178fe1e4a2ba342c24c26990a84 P eeee1578317046a7acff34fd57ab3022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b244d023c7247219f0d39a360b3e3e0" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 18:32:13.256531 13080 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 18:32:14.840667 12507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:32:14.840988 12507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:32:14.845098 12507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f77c29104bae4b95b9f3c560d0589bb9: Sending DeleteTablet for 3 replicas of tablet cb4b1178fe1e4a2ba342c24c26990a84
I20260620 18:32:14.845940 12786 tablet_service.cc:1514] Processing DeleteTablet for tablet cb4b1178fe1e4a2ba342c24c26990a84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:14 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:50920
I20260620 18:32:14.846088 12926 tablet_service.cc:1514] Processing DeleteTablet for tablet cb4b1178fe1e4a2ba342c24c26990a84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:14 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:59370
I20260620 18:32:14.846135 12646 tablet_service.cc:1514] Processing DeleteTablet for tablet cb4b1178fe1e4a2ba342c24c26990a84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:14 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:60390
I20260620 18:32:14.846581 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 12580
I20260620 18:32:14.857736 13163 tablet_replica.cc:330] T cb4b1178fe1e4a2ba342c24c26990a84 P eeee1578317046a7acff34fd57ab3022: stopping tablet replica
I20260620 18:32:14.858253 13163 raft_consensus.cc:2229] T cb4b1178fe1e4a2ba342c24c26990a84 P eeee1578317046a7acff34fd57ab3022 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:32:14.858946 13163 raft_consensus.cc:2258] T cb4b1178fe1e4a2ba342c24c26990a84 P eeee1578317046a7acff34fd57ab3022 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:14.861263 13163 ts_tablet_manager.cc:1905] T cb4b1178fe1e4a2ba342c24c26990a84 P eeee1578317046a7acff34fd57ab3022: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:14.861790 13165 tablet_replica.cc:330] T cb4b1178fe1e4a2ba342c24c26990a84 P 38da097b32ae4db9a2d2ad6f5e03d266: stopping tablet replica
I20260620 18:32:14.862191 13165 raft_consensus.cc:2229] T cb4b1178fe1e4a2ba342c24c26990a84 P 38da097b32ae4db9a2d2ad6f5e03d266 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:32:14.862332 13164 tablet_replica.cc:330] T cb4b1178fe1e4a2ba342c24c26990a84 P 5b244d023c7247219f0d39a360b3e3e0: stopping tablet replica
I20260620 18:32:14.862607 13165 raft_consensus.cc:2258] T cb4b1178fe1e4a2ba342c24c26990a84 P 38da097b32ae4db9a2d2ad6f5e03d266 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:14.863240 13164 raft_consensus.cc:2229] T cb4b1178fe1e4a2ba342c24c26990a84 P 5b244d023c7247219f0d39a360b3e3e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:32:14.863567 13164 raft_consensus.cc:2258] T cb4b1178fe1e4a2ba342c24c26990a84 P 5b244d023c7247219f0d39a360b3e3e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:14.863868 13165 ts_tablet_manager.cc:1905] T cb4b1178fe1e4a2ba342c24c26990a84 P 38da097b32ae4db9a2d2ad6f5e03d266: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:14.864547 13164 ts_tablet_manager.cc:1905] T cb4b1178fe1e4a2ba342c24c26990a84 P 5b244d023c7247219f0d39a360b3e3e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:14.869576 13163 ts_tablet_manager.cc:1918] T cb4b1178fe1e4a2ba342c24c26990a84 P eeee1578317046a7acff34fd57ab3022: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:32:14.869766 13163 log.cc:1196] T cb4b1178fe1e4a2ba342c24c26990a84 P eeee1578317046a7acff34fd57ab3022: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-1/wal/wals/cb4b1178fe1e4a2ba342c24c26990a84
I20260620 18:32:14.870252 13163 ts_tablet_manager.cc:1939] T cb4b1178fe1e4a2ba342c24c26990a84 P eeee1578317046a7acff34fd57ab3022: Deleting consensus metadata
I20260620 18:32:14.870786 13165 ts_tablet_manager.cc:1918] T cb4b1178fe1e4a2ba342c24c26990a84 P 38da097b32ae4db9a2d2ad6f5e03d266: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:32:14.870980 13165 log.cc:1196] T cb4b1178fe1e4a2ba342c24c26990a84 P 38da097b32ae4db9a2d2ad6f5e03d266: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-0/wal/wals/cb4b1178fe1e4a2ba342c24c26990a84
I20260620 18:32:14.871493 13165 ts_tablet_manager.cc:1939] T cb4b1178fe1e4a2ba342c24c26990a84 P 38da097b32ae4db9a2d2ad6f5e03d266: Deleting consensus metadata
I20260620 18:32:14.871515 12494 catalog_manager.cc:4841] TS eeee1578317046a7acff34fd57ab3022 (127.10.131.2:36507): tablet cb4b1178fe1e4a2ba342c24c26990a84 (table test-table [id=85606a0651664c65a87a632a3f0acb72]) successfully deleted
I20260620 18:32:14.872282 13164 ts_tablet_manager.cc:1918] T cb4b1178fe1e4a2ba342c24c26990a84 P 5b244d023c7247219f0d39a360b3e3e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:32:14.872429 13164 log.cc:1196] T cb4b1178fe1e4a2ba342c24c26990a84 P 5b244d023c7247219f0d39a360b3e3e0: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781980314095189-10764-0/minicluster-data/ts-2/wal/wals/cb4b1178fe1e4a2ba342c24c26990a84
I20260620 18:32:14.872610 12493 catalog_manager.cc:4841] TS 38da097b32ae4db9a2d2ad6f5e03d266 (127.10.131.1:41907): tablet cb4b1178fe1e4a2ba342c24c26990a84 (table test-table [id=85606a0651664c65a87a632a3f0acb72]) successfully deleted
I20260620 18:32:14.872936 13164 ts_tablet_manager.cc:1939] T cb4b1178fe1e4a2ba342c24c26990a84 P 5b244d023c7247219f0d39a360b3e3e0: Deleting consensus metadata
I20260620 18:32:14.873842 12494 catalog_manager.cc:4841] TS 5b244d023c7247219f0d39a360b3e3e0 (127.10.131.3:35431): tablet cb4b1178fe1e4a2ba342c24c26990a84 (table test-table [id=85606a0651664c65a87a632a3f0acb72]) successfully deleted
Jun 20 18:32:14 dist-test-slave-cbn3 krb5kdc[12464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980332, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.131.1@KRBTEST.COM
I20260620 18:32:14.881147 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:14.848384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60406 (local address 127.10.131.1:41907)
0620 18:32:14.862630 (+ 14246us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:14.862638 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:14.863195 (+ 557us) server_negotiation.cc:407] Connection header received
0620 18:32:14.863466 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:14.863471 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:14.863569 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:14.863698 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:14.865541 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:14.866728 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:14.868683 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:14.869367 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:14.869503 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:14.875361 (+ 5858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:14.875385 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:14.875406 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:14.875448 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:14.878526 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:14.879240 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:14.879246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:14.879249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:14.879374 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:14.879796 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:14.879799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:14.879801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:14.880141 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:14.880356 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:14.880609 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 18:32:14.880783 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14062,"thread_start_us":13956,"threads_started":1}
W20260620 18:32:14.881897 12711 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60406
W20260620 18:32:14.882531 10764 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 12580: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:32:14.882750 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 12580
I20260620 18:32:14.904721 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 12720
Jun 20 18:32:14 dist-test-slave-cbn3 krb5kdc[12464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980332, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.131.2@KRBTEST.COM
I20260620 18:32:14.920560 13168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:14.905522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50946 (local address 127.10.131.2:36507)
0620 18:32:14.906027 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:14.906032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:14.906048 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:32:14.906102 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:14.906105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:14.906160 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:14.906295 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:14.907647 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:14.908467 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:14.909777 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:14.910175 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:14.910277 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:14.914925 (+ 4648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:14.914940 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:14.914956 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:14.914996 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:14.917835 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:14.918494 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:14.918507 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:14.918509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:14.918645 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:14.918998 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:14.919004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:14.919007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:14.919367 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:14.919674 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:14.919995 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 18:32:14.920205 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":256,"threads_started":1}
W20260620 18:32:14.921306 12851 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50946
W20260620 18:32:14.921912 10764 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 12720: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:32:14.922058 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 12720
I20260620 18:32:14.947587 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 12860
Jun 20 18:32:14 dist-test-slave-cbn3 krb5kdc[12464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980332, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.131.3@KRBTEST.COM
I20260620 18:32:14.962790 13169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:14.948279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59392 (local address 127.10.131.3:35431)
0620 18:32:14.948783 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:14.948788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:14.948804 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:32:14.948913 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:14.948917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:14.948980 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:14.949070 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:14.950293 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:14.951084 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:14.952477 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:14.953020 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:14.953124 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:14.957507 (+ 4383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:14.957521 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:14.957537 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:14.957572 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:14.960383 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:14.961073 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:14.961080 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:14.961082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:14.961197 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:14.961501 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:14.961504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:14.961506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:14.961848 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:14.962044 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:14.962265 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 18:32:14.962427 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":247,"threads_started":1}
W20260620 18:32:14.963500 12991 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59392
W20260620 18:32:14.964120 10764 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 12860: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:32:14.964314 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 12860
I20260620 18:32:14.985859 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 12477
I20260620 18:32:14.987059 12549 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:15.102816 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 12477
2026-06-20T18:32:15Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8327 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:32:15 dist-test-slave-cbn3 krb5kdc[13173](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:32:15 dist-test-slave-cbn3 krb5kdc[13173](info): set up 2 sockets
Jun 20 18:32:15 dist-test-slave-cbn3 krb5kdc[13173](info): commencing operation
krb5kdc: starting...
W20260620 18:32:16.213810 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:32:16 dist-test-slave-cbn3 krb5kdc[13173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:32:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:32:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.131.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
I20260620 18:32:16.399861 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:39779
--webserver_interface=127.10.131.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:34043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:39779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:16.711838 13186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:16.712359 13186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:16.712539 13186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:16.720082 13186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:32:16.720238 13186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:16.720355 13186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:16.720451 13186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:32:16.720537 13186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:32:16.730352 13186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:34043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:39779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:39779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:16.731662 13186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:16.732926 13186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:16.744494 13191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:16.744596 13194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:16.744596 13192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:16.745187 13186 server_base.cc:1034] running on GCE node
I20260620 18:32:16.745986 13186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:16.747315 13186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:16.748531 13186 hybrid_clock.cc:648] HybridClock initialized: now 1781980336748493 us; error 34 us; skew 500 ppm
I20260620 18:32:16.748900 13186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:16 dist-test-slave-cbn3 krb5kdc[13173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980336, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:16.751947 13186 init.cc:377] Logged in from keytab as kudu/127.10.131.62@KRBTEST.COM (short username kudu)
I20260620 18:32:16.753437 13186 webserver.cc:460] Webserver started at http://127.10.131.62:36017/ using document root <none> and password file <none>
I20260620 18:32:16.754136 13186 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:16.754274 13186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:16.754583 13186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:16.757520 13186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "6d71015c663e4964b93953ee4d2bb85c"
format_stamp: "Formatted at 2026-06-20 18:32:16 on dist-test-slave-cbn3"
server_key: "d2694fbacc336bd0d7b39c7c059db326"
server_key_iv: "184e8c6a04ab149ca5abb79ca7204b96"
server_key_version: "encryptionkey@0"
I20260620 18:32:16.758246 13186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "6d71015c663e4964b93953ee4d2bb85c"
format_stamp: "Formatted at 2026-06-20 18:32:16 on dist-test-slave-cbn3"
server_key: "d2694fbacc336bd0d7b39c7c059db326"
server_key_iv: "184e8c6a04ab149ca5abb79ca7204b96"
server_key_version: "encryptionkey@0"
I20260620 18:32:16.763175 13186 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:32:16.766536 13201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:16.767817 13186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:32:16.767999 13186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "6d71015c663e4964b93953ee4d2bb85c"
format_stamp: "Formatted at 2026-06-20 18:32:16 on dist-test-slave-cbn3"
server_key: "d2694fbacc336bd0d7b39c7c059db326"
server_key_iv: "184e8c6a04ab149ca5abb79ca7204b96"
server_key_version: "encryptionkey@0"
I20260620 18:32:16.768191 13186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:16.788920 13186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:16.793310 13186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:16.793608 13186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:16.811014 13186 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:39779
I20260620 18:32:16.811065 13252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:39779 every 8 connection(s)
I20260620 18:32:16.812333 13186 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:32:16.815985 13253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:16.816711 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 13186
I20260620 18:32:16.816967 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:32:16.817621 10764 external_mini_cluster.cc:1428] Setting key f8436590e61941fafd99b6562fb7990c
I20260620 18:32:16.830443 13253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d71015c663e4964b93953ee4d2bb85c: Bootstrap starting.
Jun 20 18:32:16 dist-test-slave-cbn3 krb5kdc[13173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:32:16.833763 13253 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d71015c663e4964b93953ee4d2bb85c: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:16.835175 13253 log.cc:824] T 00000000000000000000000000000000 P 6d71015c663e4964b93953ee4d2bb85c: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:16.838647 13253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d71015c663e4964b93953ee4d2bb85c: No bootstrap required, opened a new log
I20260620 18:32:16.841939 13256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:16.819682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48550 (local address 127.10.131.62:39779)
0620 18:32:16.820499 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:16.820520 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:16.820576 (+ 56us) server_negotiation.cc:407] Connection header received
0620 18:32:16.822715 (+ 2139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:16.822742 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:16.823246 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:16.823706 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:16.824958 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:16.826972 (+ 2014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:16.828306 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:16.829391 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:16.829584 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:16.833339 (+ 3755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:16.833352 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:16.833370 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:16.833426 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:16.837328 (+ 3902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:16.838055 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:16.838064 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:16.838071 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:16.838285 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:16.838638 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:16.838645 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:16.838648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:16.839354 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:16.839652 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:16.839998 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 18:32:16.840407 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":308,"threads_started":1}
I20260620 18:32:16.847998 13253 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d71015c663e4964b93953ee4d2bb85c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d71015c663e4964b93953ee4d2bb85c" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 39779 } }
I20260620 18:32:16.848342 13253 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d71015c663e4964b93953ee4d2bb85c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:16.848461 13253 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d71015c663e4964b93953ee4d2bb85c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d71015c663e4964b93953ee4d2bb85c, State: Initialized, Role: FOLLOWER
I20260620 18:32:16.849112 13253 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d71015c663e4964b93953ee4d2bb85c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d71015c663e4964b93953ee4d2bb85c" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 39779 } }
I20260620 18:32:16.849445 13253 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d71015c663e4964b93953ee4d2bb85c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:32:16.849579 13253 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d71015c663e4964b93953ee4d2bb85c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:32:16.849756 13253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d71015c663e4964b93953ee4d2bb85c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:16.851795 13253 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d71015c663e4964b93953ee4d2bb85c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d71015c663e4964b93953ee4d2bb85c" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 39779 } }
I20260620 18:32:16.852262 13253 leader_election.cc:302] T 00000000000000000000000000000000 P 6d71015c663e4964b93953ee4d2bb85c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d71015c663e4964b93953ee4d2bb85c; no voters:
I20260620 18:32:16.852824 13253 leader_election.cc:288] T 00000000000000000000000000000000 P 6d71015c663e4964b93953ee4d2bb85c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:32:16.853000 13258 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d71015c663e4964b93953ee4d2bb85c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:16.853672 13258 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d71015c663e4964b93953ee4d2bb85c [term 1 LEADER]: Becoming Leader. State: Replica: 6d71015c663e4964b93953ee4d2bb85c, State: Running, Role: LEADER
I20260620 18:32:16.854156 13258 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d71015c663e4964b93953ee4d2bb85c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d71015c663e4964b93953ee4d2bb85c" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 39779 } }
I20260620 18:32:16.854910 13253 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d71015c663e4964b93953ee4d2bb85c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:32:16.856698 13260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d71015c663e4964b93953ee4d2bb85c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d71015c663e4964b93953ee4d2bb85c. Latest consensus state: current_term: 1 leader_uuid: "6d71015c663e4964b93953ee4d2bb85c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d71015c663e4964b93953ee4d2bb85c" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 39779 } } }
I20260620 18:32:16.857054 13260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d71015c663e4964b93953ee4d2bb85c [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:16.857795 13259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d71015c663e4964b93953ee4d2bb85c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d71015c663e4964b93953ee4d2bb85c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d71015c663e4964b93953ee4d2bb85c" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 39779 } } }
I20260620 18:32:16.858039 13264 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:32:16.858093 13259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d71015c663e4964b93953ee4d2bb85c [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:16.864490 13264 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:32:16.873045 13264 catalog_manager.cc:1347] Generated new cluster ID: f71fb8d1b7c341c292cc01c9d7bdc92d
I20260620 18:32:16.873175 13264 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:32:16.882902 13264 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:32:16.884215 13264 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:32:16.899595 13264 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d71015c663e4964b93953ee4d2bb85c: Generated new TSK 0
I20260620 18:32:16.900413 13264 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.131.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab.
Entry for principal kudu/127.10.131.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab.
I20260620 18:32:16.942462 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.1:0
--local_ip_for_outbound_sockets=127.10.131.1
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:39779
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:34043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:17.258021 13279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:17.258279 13279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:17.258370 13279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:17.265556 13279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:17.265661 13279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:17.265779 13279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.1
I20260620 18:32:17.277423 13279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:34043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:39779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.1
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:17.278968 13279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:17.280577 13279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:17.293836 13285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:17.293846 13287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:17.293939 13284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:17.294886 13279 server_base.cc:1034] running on GCE node
I20260620 18:32:17.295535 13279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:17.296440 13279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:17.297657 13279 hybrid_clock.cc:648] HybridClock initialized: now 1781980337297600 us; error 54 us; skew 500 ppm
I20260620 18:32:17.297961 13279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:17 dist-test-slave-cbn3 krb5kdc[13173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980337, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:17.301046 13279 init.cc:377] Logged in from keytab as kudu/127.10.131.1@KRBTEST.COM (short username kudu)
I20260620 18:32:17.302506 13279 webserver.cc:460] Webserver started at http://127.10.131.1:39059/ using document root <none> and password file <none>
I20260620 18:32:17.303177 13279 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:17.303319 13279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:17.303624 13279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:17.306460 13279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-0/data/instance:
uuid: "a83ff52fe4dc4f7392110ceaec2075ae"
format_stamp: "Formatted at 2026-06-20 18:32:17 on dist-test-slave-cbn3"
server_key: "42ab07e30e3fd7df95a873935f7639cb"
server_key_iv: "d530d58b1229780363c98580fe689507"
server_key_version: "encryptionkey@0"
I20260620 18:32:17.307250 13279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance:
uuid: "a83ff52fe4dc4f7392110ceaec2075ae"
format_stamp: "Formatted at 2026-06-20 18:32:17 on dist-test-slave-cbn3"
server_key: "42ab07e30e3fd7df95a873935f7639cb"
server_key_iv: "d530d58b1229780363c98580fe689507"
server_key_version: "encryptionkey@0"
I20260620 18:32:17.312808 13279 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:32:17.316431 13294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:17.317762 13279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:32:17.317930 13279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-0/wal
uuid: "a83ff52fe4dc4f7392110ceaec2075ae"
format_stamp: "Formatted at 2026-06-20 18:32:17 on dist-test-slave-cbn3"
server_key: "42ab07e30e3fd7df95a873935f7639cb"
server_key_iv: "d530d58b1229780363c98580fe689507"
server_key_version: "encryptionkey@0"
I20260620 18:32:17.318137 13279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:17.344169 13279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:17.348026 13279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:17.348321 13279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:17.349316 13279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:17.350585 13279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:17.350689 13279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:17.350831 13279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:17.350901 13279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:17.382916 13279 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.1:46581
I20260620 18:32:17.382962 13406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.1:46581 every 8 connection(s)
I20260620 18:32:17.384168 13279 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
I20260620 18:32:17.389856 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 13279
I20260620 18:32:17.390079 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance
I20260620 18:32:17.390687 10764 external_mini_cluster.cc:1428] Setting key 68812dc92415fdf5bf8259b9755c13e1
Jun 20 18:32:17 dist-test-slave-cbn3 krb5kdc[13173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980337, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:32:17.405911 13409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:17.386576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:34445 (local address 127.10.131.62:39779)
0620 18:32:17.386994 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:17.386998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:17.387892 (+ 894us) server_negotiation.cc:407] Connection header received
0620 18:32:17.389775 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:17.389780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:17.389865 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:17.389982 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:17.391492 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:17.392545 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:17.395031 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:17.395421 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:17.395516 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:17.400184 (+ 4668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:17.400189 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:17.400192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:17.400224 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:17.402821 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:17.403566 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:17.403571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:17.403573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:17.403688 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:17.404109 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:17.404132 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:17.404135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:17.404402 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:17.404557 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:17.405322 (+ 765us) server_negotiation.cc:299] Negotiation successful
0620 18:32:17.405546 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":204,"threads_started":1}
I20260620 18:32:17.406864 13410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:17.387078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:39779 (local address 127.10.131.1:34445)
0620 18:32:17.387728 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:17.387769 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:17.389514 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:17.390246 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:17.390254 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:17.390732 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:17.391289 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:17.391305 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:17.392743 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:17.392749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:17.394881 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:17.394906 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:17.395643 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:17.395652 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:17.395985 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:17.396748 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:17.396782 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:17.399929 (+ 3147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:17.402973 (+ 3044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:17.402982 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:17.402999 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:17.403433 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:17.403798 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:17.403803 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:17.403807 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:17.403990 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:17.404533 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:17.404541 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:17.404955 (+ 414us) client_negotiation.cc:769] Sending connection context
0620 18:32:17.405234 (+ 279us) client_negotiation.cc:240] Negotiation successful
0620 18:32:17.405758 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":244,"threads_started":1}
I20260620 18:32:17.408456 13407 heartbeater.cc:347] Connected to a master server at 127.10.131.62:39779
I20260620 18:32:17.408816 13407 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.10.131.2@KRBTEST.COM; defaulting to no policy
I20260620 18:32:17.409679 13407 heartbeater.cc:511] Master 127.10.131.62:39779 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.2@KRBTEST.COM" created.
I20260620 18:32:17.411823 13218 ts_manager.cc:194] Registered new tserver with Master: a83ff52fe4dc4f7392110ceaec2075ae (127.10.131.1:46581)
I20260620 18:32:17.413367 13218 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.1@KRBTEST.COM'} at 127.10.131.1:34445
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab.
Entry for principal kudu/127.10.131.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab.
I20260620 18:32:17.433501 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.2:0
--local_ip_for_outbound_sockets=127.10.131.2
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:39779
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:34043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:17.751405 13413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:17.751660 13413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:17.751753 13413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:17.759150 13413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:17.759258 13413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:17.759370 13413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.2
I20260620 18:32:17.769589 13413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:34043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:39779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.2
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:17.770879 13413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:17.772295 13413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:17.785130 13418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:17.785095 13419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:17.785287 13421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:17.786547 13413 server_base.cc:1034] running on GCE node
I20260620 18:32:17.787165 13413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:17.788017 13413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:17.789227 13413 hybrid_clock.cc:648] HybridClock initialized: now 1781980337789178 us; error 38 us; skew 500 ppm
I20260620 18:32:17.789489 13413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:17 dist-test-slave-cbn3 krb5kdc[13173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980337, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:17.792650 13413 init.cc:377] Logged in from keytab as kudu/127.10.131.2@KRBTEST.COM (short username kudu)
I20260620 18:32:17.794175 13413 webserver.cc:460] Webserver started at http://127.10.131.2:38803/ using document root <none> and password file <none>
I20260620 18:32:17.794860 13413 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:17.794958 13413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:17.795297 13413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:17.798157 13413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-1/data/instance:
uuid: "c47944eec30f48ab8b903a11b22fe6b2"
format_stamp: "Formatted at 2026-06-20 18:32:17 on dist-test-slave-cbn3"
server_key: "effe745b4166996621d2a96c6b8a4959"
server_key_iv: "d4d30b2d3a0a38e6f5d26f83c0f941f1"
server_key_version: "encryptionkey@0"
I20260620 18:32:17.798919 13413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance:
uuid: "c47944eec30f48ab8b903a11b22fe6b2"
format_stamp: "Formatted at 2026-06-20 18:32:17 on dist-test-slave-cbn3"
server_key: "effe745b4166996621d2a96c6b8a4959"
server_key_iv: "d4d30b2d3a0a38e6f5d26f83c0f941f1"
server_key_version: "encryptionkey@0"
I20260620 18:32:17.804059 13413 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 18:32:17.807508 13428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:17.808806 13413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:32:17.809015 13413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-1/wal
uuid: "c47944eec30f48ab8b903a11b22fe6b2"
format_stamp: "Formatted at 2026-06-20 18:32:17 on dist-test-slave-cbn3"
server_key: "effe745b4166996621d2a96c6b8a4959"
server_key_iv: "d4d30b2d3a0a38e6f5d26f83c0f941f1"
server_key_version: "encryptionkey@0"
I20260620 18:32:17.809214 13413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:17.825052 13413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:17.829203 13413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:17.829478 13413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:17.830377 13413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:17.831697 13413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:17.831795 13413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:17.831959 13413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:17.832031 13413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:17.866299 13413 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.2:36731
I20260620 18:32:17.866331 13540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.2:36731 every 8 connection(s)
I20260620 18:32:17.867556 13413 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
I20260620 18:32:17.870559 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 13413
I20260620 18:32:17.870750 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance
I20260620 18:32:17.871387 10764 external_mini_cluster.cc:1428] Setting key c5d45e716b4cb34c0bf8834641a06373
Jun 20 18:32:17 dist-test-slave-cbn3 krb5kdc[13173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980337, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:32:17.890377 13409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:17.870016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:39641 (local address 127.10.131.62:39779)
0620 18:32:17.870202 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:17.870206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:17.871452 (+ 1246us) server_negotiation.cc:407] Connection header received
0620 18:32:17.873459 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:17.873463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:17.873524 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:17.873601 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:17.875630 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:17.876342 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:17.878824 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:17.879555 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:17.879689 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:17.883030 (+ 3341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:17.883039 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:17.883047 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:17.883091 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:17.886993 (+ 3902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:17.887763 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:17.887772 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:17.887778 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:17.887960 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:17.888436 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:17.888441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:17.888445 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:17.888818 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:17.889019 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:17.889746 (+ 727us) server_negotiation.cc:299] Negotiation successful
0620 18:32:17.890000 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
WARNING: no policy specified for kudu/127.10.131.3@KRBTEST.COM; defaulting to no policy
I20260620 18:32:17.891083 13543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:17.870447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:39779 (local address 127.10.131.2:39641)
0620 18:32:17.871286 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:17.871332 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:17.873179 (+ 1847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:17.873887 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:17.873899 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:17.874659 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:17.875462 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:17.875487 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:17.876504 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:17.876511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:17.878676 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:17.878693 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:17.879767 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:17.879772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:17.879970 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:17.880628 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:17.880650 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:17.882809 (+ 2159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:17.887186 (+ 4377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:17.887193 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:17.887205 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:17.887624 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:17.888087 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:17.888111 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:17.888114 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:17.888302 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:17.888993 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:17.889001 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:17.889356 (+ 355us) client_negotiation.cc:769] Sending connection context
0620 18:32:17.889599 (+ 243us) client_negotiation.cc:240] Negotiation successful
0620 18:32:17.889988 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":351,"threads_started":1}
I20260620 18:32:17.892799 13541 heartbeater.cc:347] Connected to a master server at 127.10.131.62:39779
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.3@KRBTEST.COM" created.
I20260620 18:32:17.893132 13541 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:17.893884 13541 heartbeater.cc:511] Master 127.10.131.62:39779 requested a full tablet report, sending...
I20260620 18:32:17.895318 13218 ts_manager.cc:194] Registered new tserver with Master: c47944eec30f48ab8b903a11b22fe6b2 (127.10.131.2:36731)
I20260620 18:32:17.896100 13218 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.2@KRBTEST.COM'} at 127.10.131.2:39641
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab.
Entry for principal kudu/127.10.131.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab.
I20260620 18:32:17.913975 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.3:0
--local_ip_for_outbound_sockets=127.10.131.3
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:39779
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:34043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:18.232431 13546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:18.232684 13546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:18.232774 13546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:18.239957 13546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:18.240058 13546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:18.240180 13546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.3
I20260620 18:32:18.250249 13546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:34043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:39779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.3
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:18.251391 13546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:18.252588 13546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:18.265506 13554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:18.265583 13551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:18.265637 13552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:18.266304 13546 server_base.cc:1034] running on GCE node
I20260620 18:32:18.266868 13546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:18.267738 13546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:18.268926 13546 hybrid_clock.cc:648] HybridClock initialized: now 1781980338268887 us; error 53 us; skew 500 ppm
I20260620 18:32:18.269227 13546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:18 dist-test-slave-cbn3 krb5kdc[13173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980338, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:18.272440 13546 init.cc:377] Logged in from keytab as kudu/127.10.131.3@KRBTEST.COM (short username kudu)
I20260620 18:32:18.273949 13546 webserver.cc:460] Webserver started at http://127.10.131.3:41939/ using document root <none> and password file <none>
I20260620 18:32:18.274583 13546 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:18.274688 13546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:18.274940 13546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:18.277875 13546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-2/data/instance:
uuid: "c0d1dcaca6534daaaffd072fa2a028ef"
format_stamp: "Formatted at 2026-06-20 18:32:18 on dist-test-slave-cbn3"
server_key: "99b29d615800407a0ceb790c432ae93e"
server_key_iv: "377de22a53f8c01e01cf192f1d7da40f"
server_key_version: "encryptionkey@0"
I20260620 18:32:18.278607 13546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance:
uuid: "c0d1dcaca6534daaaffd072fa2a028ef"
format_stamp: "Formatted at 2026-06-20 18:32:18 on dist-test-slave-cbn3"
server_key: "99b29d615800407a0ceb790c432ae93e"
server_key_iv: "377de22a53f8c01e01cf192f1d7da40f"
server_key_version: "encryptionkey@0"
I20260620 18:32:18.283823 13546 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 18:32:18.287295 13561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:18.288667 13546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 18:32:18.288802 13546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-2/wal
uuid: "c0d1dcaca6534daaaffd072fa2a028ef"
format_stamp: "Formatted at 2026-06-20 18:32:18 on dist-test-slave-cbn3"
server_key: "99b29d615800407a0ceb790c432ae93e"
server_key_iv: "377de22a53f8c01e01cf192f1d7da40f"
server_key_version: "encryptionkey@0"
I20260620 18:32:18.289062 13546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:18.304692 13546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:18.309199 13546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:18.309470 13546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:18.310391 13546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:18.311671 13546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:18.311790 13546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:18.311946 13546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:18.312018 13546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:18.346624 13546 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.3:36383
I20260620 18:32:18.346680 13673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.3:36383 every 8 connection(s)
I20260620 18:32:18.347906 13546 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
I20260620 18:32:18.351469 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 13546
I20260620 18:32:18.351671 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance
I20260620 18:32:18.352190 10764 external_mini_cluster.cc:1428] Setting key b398b74b722a6a5026c153266900c314
Jun 20 18:32:18 dist-test-slave-cbn3 krb5kdc[13173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980338, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:32:18.367575 13409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:18.350410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:58099 (local address 127.10.131.62:39779)
0620 18:32:18.350593 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:18.350597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:18.351773 (+ 1176us) server_negotiation.cc:407] Connection header received
0620 18:32:18.353902 (+ 2129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:18.353907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:18.353992 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:18.354109 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:18.355665 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:18.356510 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:18.358033 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:18.358449 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:18.358589 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:18.361946 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:18.361951 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:18.361954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:18.361986 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:18.364558 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:18.365366 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:18.365372 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:18.365375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:18.365499 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:18.365952 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:18.365956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:18.365960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:18.366222 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:18.366377 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:18.367007 (+ 630us) server_negotiation.cc:299] Negotiation successful
0620 18:32:18.367236 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:32:18.368496 13676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:18.350838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:39779 (local address 127.10.131.3:58099)
0620 18:32:18.351577 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:18.351626 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:18.353645 (+ 2019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:18.354341 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:18.354349 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:18.354837 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:18.355500 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:18.355517 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:18.356657 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:18.356667 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:18.357891 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:18.357902 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:18.358668 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:18.358678 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:18.358927 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:18.359577 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:18.359598 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:18.361746 (+ 2148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:18.364744 (+ 2998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:18.364752 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:18.364767 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:18.365244 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:18.365613 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:18.365619 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:18.365623 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:18.365818 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:18.366349 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:18.366357 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:18.366674 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 18:32:18.366922 (+ 248us) client_negotiation.cc:240] Negotiation successful
0620 18:32:18.367418 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":323,"threads_started":1}
I20260620 18:32:18.369899 13674 heartbeater.cc:347] Connected to a master server at 127.10.131.62:39779
I20260620 18:32:18.370206 13674 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:18.370903 13674 heartbeater.cc:511] Master 127.10.131.62:39779 requested a full tablet report, sending...
I20260620 18:32:18.372306 13218 ts_manager.cc:194] Registered new tserver with Master: c0d1dcaca6534daaaffd072fa2a028ef (127.10.131.3:36383)
I20260620 18:32:18.373276 13218 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.3@KRBTEST.COM'} at 127.10.131.3:58099
I20260620 18:32:18.379145 10764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 18:32:18.412304 13409 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:32:18.410004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48554 (local address 127.10.131.62:39779)
0620 18:32:18.410261 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:18.410270 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:18.410595 (+ 325us) server_negotiation.cc:407] Connection header received
0620 18:32:18.410734 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:18.410738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:18.410804 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:18.410892 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:18.412038 (+ 1146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48554: BlockingRecv error: recv got EOF from 127.0.0.1:48554 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 18:32:18.415501 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 13279
I20260620 18:32:18.416412 13407 heartbeater.cc:503] Master 127.10.131.62:39779 was elected leader, sending a full tablet report...
W20260620 18:32:18.418507 10764 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 13279: Not authorized: Client connection negotiation failed: client connection to 127.10.131.1:46581: server requires authentication, but client does not have Kerberos credentials available
I20260620 18:32:18.418684 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 13279
I20260620 18:32:18.433987 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 13413
W20260620 18:32:18.436407 10764 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 13413: Not authorized: Client connection negotiation failed: client connection to 127.10.131.2:36731: server requires authentication, but client does not have Kerberos credentials available
I20260620 18:32:18.436584 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 13413
I20260620 18:32:18.451615 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 13546
W20260620 18:32:18.454761 10764 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 13546: Not authorized: Client connection negotiation failed: client connection to 127.10.131.3:36383: server requires authentication, but client does not have Kerberos credentials available
I20260620 18:32:18.454890 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 13546
I20260620 18:32:18.470477 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 13186
I20260620 18:32:18.471763 13248 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:18.572554 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 13186
2026-06-20T18:32:18Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3466 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:32:18 dist-test-slave-cbn3 krb5kdc[13691](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:32:18 dist-test-slave-cbn3 krb5kdc[13691](info): set up 2 sockets
Jun 20 18:32:18 dist-test-slave-cbn3 krb5kdc[13691](info): commencing operation
krb5kdc: starting...
W20260620 18:32:19.662230 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:32:19 dist-test-slave-cbn3 krb5kdc[13691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:32:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:32:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.131.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
I20260620 18:32:19.852252 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:34871
--webserver_interface=127.10.131.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:44011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:34871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:20.163458 13704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:20.163849 13704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:20.164017 13704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:20.171263 13704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:32:20.171413 13704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:20.171528 13704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:20.171631 13704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:32:20.171708 13704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:32:20.181422 13704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:44011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:34871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:34871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:20.182689 13704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:20.183938 13704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:20.195238 13709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:20.195387 13712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:20.195729 13710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:20.196333 13704 server_base.cc:1034] running on GCE node
I20260620 18:32:20.197245 13704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:20.198766 13704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:20.200017 13704 hybrid_clock.cc:648] HybridClock initialized: now 1781980340199975 us; error 57 us; skew 500 ppm
I20260620 18:32:20.200397 13704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:20 dist-test-slave-cbn3 krb5kdc[13691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980340, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:20.208539 13704 init.cc:377] Logged in from keytab as kudu/127.10.131.62@KRBTEST.COM (short username kudu)
I20260620 18:32:20.210125 13704 webserver.cc:460] Webserver started at http://127.10.131.62:38149/ using document root <none> and password file <none>
I20260620 18:32:20.210830 13704 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:20.210978 13704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:20.211304 13704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:20.214466 13704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "169a7997657f43c6af817d6fa5982b73"
format_stamp: "Formatted at 2026-06-20 18:32:20 on dist-test-slave-cbn3"
server_key: "a465a3befeae2b56b2a556d08e632835"
server_key_iv: "8da5c32255138a51c5771071485b052f"
server_key_version: "encryptionkey@0"
I20260620 18:32:20.215330 13704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "169a7997657f43c6af817d6fa5982b73"
format_stamp: "Formatted at 2026-06-20 18:32:20 on dist-test-slave-cbn3"
server_key: "a465a3befeae2b56b2a556d08e632835"
server_key_iv: "8da5c32255138a51c5771071485b052f"
server_key_version: "encryptionkey@0"
I20260620 18:32:20.220330 13704 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:32:20.223706 13719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:20.225116 13704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:32:20.225296 13704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "169a7997657f43c6af817d6fa5982b73"
format_stamp: "Formatted at 2026-06-20 18:32:20 on dist-test-slave-cbn3"
server_key: "a465a3befeae2b56b2a556d08e632835"
server_key_iv: "8da5c32255138a51c5771071485b052f"
server_key_version: "encryptionkey@0"
I20260620 18:32:20.225512 13704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:20.250618 13704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:20.255103 13704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:20.255420 13704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:20.273818 13704 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:34871
I20260620 18:32:20.273852 13770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:34871 every 8 connection(s)
I20260620 18:32:20.275177 13704 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:32:20.278817 13771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:20.279739 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 13704
I20260620 18:32:20.279978 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:32:20.280627 10764 external_mini_cluster.cc:1428] Setting key 8e4f8994d484017c988f7cfaa449021f
I20260620 18:32:20.290956 13771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 169a7997657f43c6af817d6fa5982b73: Bootstrap starting.
I20260620 18:32:20.294214 13771 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 169a7997657f43c6af817d6fa5982b73: Neither blocks nor log segments found. Creating new log.
Jun 20 18:32:20 dist-test-slave-cbn3 krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:32:20.295253 13771 log.cc:824] T 00000000000000000000000000000000 P 169a7997657f43c6af817d6fa5982b73: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:20.298131 13771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 169a7997657f43c6af817d6fa5982b73: No bootstrap required, opened a new log
I20260620 18:32:20.302793 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:20.282554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58008 (local address 127.10.131.62:34871)
0620 18:32:20.283347 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:20.283359 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:20.283428 (+ 69us) server_negotiation.cc:407] Connection header received
0620 18:32:20.285051 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:20.285072 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:20.285468 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:20.285864 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:20.287377 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:20.288698 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:20.289979 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:20.290656 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:20.290852 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:20.295420 (+ 4568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:20.295430 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:20.295446 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:20.295488 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:20.298657 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:20.299357 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:20.299373 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:20.299380 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:20.299543 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:20.299958 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:20.299965 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:20.299968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:20.300552 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:20.300913 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:20.301365 (+ 452us) server_negotiation.cc:299] Negotiation successful
0620 18:32:20.301794 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":363,"threads_started":1}
I20260620 18:32:20.306142 13771 raft_consensus.cc:348] T 00000000000000000000000000000000 P 169a7997657f43c6af817d6fa5982b73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "169a7997657f43c6af817d6fa5982b73" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34871 } }
I20260620 18:32:20.306504 13771 raft_consensus.cc:374] T 00000000000000000000000000000000 P 169a7997657f43c6af817d6fa5982b73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:20.306613 13771 raft_consensus.cc:729] T 00000000000000000000000000000000 P 169a7997657f43c6af817d6fa5982b73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 169a7997657f43c6af817d6fa5982b73, State: Initialized, Role: FOLLOWER
I20260620 18:32:20.306784 10764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:32:20.307235 13771 consensus_queue.cc:260] T 00000000000000000000000000000000 P 169a7997657f43c6af817d6fa5982b73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "169a7997657f43c6af817d6fa5982b73" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34871 } }
I20260620 18:32:20.307453 13771 raft_consensus.cc:388] T 00000000000000000000000000000000 P 169a7997657f43c6af817d6fa5982b73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:32:20.307577 13771 raft_consensus.cc:482] T 00000000000000000000000000000000 P 169a7997657f43c6af817d6fa5982b73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:32:20.307749 13771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 169a7997657f43c6af817d6fa5982b73 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:20.310011 13771 raft_consensus.cc:504] T 00000000000000000000000000000000 P 169a7997657f43c6af817d6fa5982b73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "169a7997657f43c6af817d6fa5982b73" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34871 } }
I20260620 18:32:20.310485 13771 leader_election.cc:302] T 00000000000000000000000000000000 P 169a7997657f43c6af817d6fa5982b73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 169a7997657f43c6af817d6fa5982b73; no voters:
I20260620 18:32:20.311131 13771 leader_election.cc:288] T 00000000000000000000000000000000 P 169a7997657f43c6af817d6fa5982b73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:32:20.311236 13776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 169a7997657f43c6af817d6fa5982b73 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:20.313220 13776 raft_consensus.cc:686] T 00000000000000000000000000000000 P 169a7997657f43c6af817d6fa5982b73 [term 1 LEADER]: Becoming Leader. State: Replica: 169a7997657f43c6af817d6fa5982b73, State: Running, Role: LEADER
I20260620 18:32:20.313588 13771 sys_catalog.cc:564] T 00000000000000000000000000000000 P 169a7997657f43c6af817d6fa5982b73 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:32:20.313791 13776 consensus_queue.cc:237] T 00000000000000000000000000000000 P 169a7997657f43c6af817d6fa5982b73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "169a7997657f43c6af817d6fa5982b73" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34871 } }
I20260620 18:32:20.316715 13777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 169a7997657f43c6af817d6fa5982b73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "169a7997657f43c6af817d6fa5982b73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "169a7997657f43c6af817d6fa5982b73" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34871 } } }
I20260620 18:32:20.317090 13777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 169a7997657f43c6af817d6fa5982b73 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:20.317627 13778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 169a7997657f43c6af817d6fa5982b73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 169a7997657f43c6af817d6fa5982b73. Latest consensus state: current_term: 1 leader_uuid: "169a7997657f43c6af817d6fa5982b73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "169a7997657f43c6af817d6fa5982b73" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34871 } } }
I20260620 18:32:20.317912 13778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 169a7997657f43c6af817d6fa5982b73 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:20.335129 13784 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:32:20.341884 13784 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:32:20.355661 13784 catalog_manager.cc:1347] Generated new cluster ID: a2e1d6468d954d05adb4632ee27b8ba0
I20260620 18:32:20.355783 13784 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:32:20.374904 13784 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:32:20.376219 13784 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:32:20.387575 13784 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 169a7997657f43c6af817d6fa5982b73: Generated new TSK 0
I20260620 18:32:20.388540 13784 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:32:20.390065 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:20.371314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58012 (local address 127.10.131.62:34871)
0620 18:32:20.371490 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:20.371496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:20.381573 (+ 10077us) server_negotiation.cc:407] Connection header received
0620 18:32:20.381803 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:20.381809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:20.381892 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:20.382003 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:20.383213 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:20.384386 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:20.386016 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:20.386698 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:20.386828 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:20.388223 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:20.388230 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:20.388235 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:32:20.389104 (+ 869us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:20.389191 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:32:20.389238 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:20.389368 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:20.389610 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 18:32:20.389714 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:32:20.397712 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 13704
I20260620 18:32:20.398922 13766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:20.485965 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 13704
2026-06-20T18:32:20Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1905 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:32:20 dist-test-slave-cbn3 krb5kdc[13805](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:32:20 dist-test-slave-cbn3 krb5kdc[13805](info): set up 2 sockets
Jun 20 18:32:20 dist-test-slave-cbn3 krb5kdc[13805](info): commencing operation
krb5kdc: starting...
W20260620 18:32:21.565734 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.003s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:32:21 dist-test-slave-cbn3 krb5kdc[13805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:32:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:32:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.131.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
I20260620 18:32:21.760599 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:39993
--webserver_interface=127.10.131.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:42789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:39993
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:22.076601 13818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:22.076917 13818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:22.077065 13818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:22.084247 13818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:32:22.084352 13818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:22.084473 13818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:22.084578 13818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:32:22.084658 13818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:32:22.094442 13818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:42789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:39993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:39993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:22.095734 13818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:22.097038 13818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:22.108527 13824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:22.108711 13826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:22.108712 13823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:22.109643 13818 server_base.cc:1034] running on GCE node
I20260620 18:32:22.110417 13818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:22.111704 13818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:22.112934 13818 hybrid_clock.cc:648] HybridClock initialized: now 1781980342112884 us; error 28 us; skew 500 ppm
I20260620 18:32:22.113220 13818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:22 dist-test-slave-cbn3 krb5kdc[13805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980342, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:22.116199 13818 init.cc:377] Logged in from keytab as kudu/127.10.131.62@KRBTEST.COM (short username kudu)
I20260620 18:32:22.117700 13818 webserver.cc:460] Webserver started at http://127.10.131.62:38733/ using document root <none> and password file <none>
I20260620 18:32:22.118373 13818 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:22.118470 13818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:22.118707 13818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:22.121563 13818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "af2a6b0d7f6748369de043ce1aadae4e"
format_stamp: "Formatted at 2026-06-20 18:32:22 on dist-test-slave-cbn3"
server_key: "64435d3e09205b38916de60248966331"
server_key_iv: "95a903c984c6c32ae49fce745d044a6c"
server_key_version: "encryptionkey@0"
I20260620 18:32:22.122252 13818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "af2a6b0d7f6748369de043ce1aadae4e"
format_stamp: "Formatted at 2026-06-20 18:32:22 on dist-test-slave-cbn3"
server_key: "64435d3e09205b38916de60248966331"
server_key_iv: "95a903c984c6c32ae49fce745d044a6c"
server_key_version: "encryptionkey@0"
I20260620 18:32:22.127126 13818 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:32:22.130437 13833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:22.131878 13818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:32:22.132005 13818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "af2a6b0d7f6748369de043ce1aadae4e"
format_stamp: "Formatted at 2026-06-20 18:32:22 on dist-test-slave-cbn3"
server_key: "64435d3e09205b38916de60248966331"
server_key_iv: "95a903c984c6c32ae49fce745d044a6c"
server_key_version: "encryptionkey@0"
I20260620 18:32:22.132155 13818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:22.157696 13818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:22.161947 13818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:22.162171 13818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:22.180096 13818 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:39993
I20260620 18:32:22.180138 13884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:39993 every 8 connection(s)
I20260620 18:32:22.181406 13818 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:32:22.184821 13885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:22.189544 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 13818
I20260620 18:32:22.189765 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:32:22.190603 10764 external_mini_cluster.cc:1428] Setting key 4e697714230a7112bb47cc2862bc491b
I20260620 18:32:22.196097 13885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af2a6b0d7f6748369de043ce1aadae4e: Bootstrap starting.
I20260620 18:32:22.199589 13885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af2a6b0d7f6748369de043ce1aadae4e: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:22.200743 13885 log.cc:824] T 00000000000000000000000000000000 P af2a6b0d7f6748369de043ce1aadae4e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:22.204236 13885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af2a6b0d7f6748369de043ce1aadae4e: No bootstrap required, opened a new log
Jun 20 18:32:22 dist-test-slave-cbn3 krb5kdc[13805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:32:22.214633 13888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:22.192554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46704 (local address 127.10.131.62:39993)
0620 18:32:22.193413 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:22.193424 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:22.193464 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:32:22.195080 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:22.195102 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:22.195466 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:22.195764 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:22.197415 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:22.198645 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:22.200382 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:22.201070 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:22.201190 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:22.205918 (+ 4728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:22.205930 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:22.205953 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:22.206006 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:22.210040 (+ 4034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:22.210713 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:22.210727 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:22.210736 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:22.210963 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:22.211288 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:22.211293 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:22.211296 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:22.212074 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:22.212366 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:22.212701 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 18:32:22.213226 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":421,"threads_started":1}
I20260620 18:32:22.215159 13885 raft_consensus.cc:348] T 00000000000000000000000000000000 P af2a6b0d7f6748369de043ce1aadae4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2a6b0d7f6748369de043ce1aadae4e" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 39993 } }
I20260620 18:32:22.215550 13885 raft_consensus.cc:374] T 00000000000000000000000000000000 P af2a6b0d7f6748369de043ce1aadae4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:22.215664 13885 raft_consensus.cc:729] T 00000000000000000000000000000000 P af2a6b0d7f6748369de043ce1aadae4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af2a6b0d7f6748369de043ce1aadae4e, State: Initialized, Role: FOLLOWER
I20260620 18:32:22.216296 13885 consensus_queue.cc:260] T 00000000000000000000000000000000 P af2a6b0d7f6748369de043ce1aadae4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2a6b0d7f6748369de043ce1aadae4e" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 39993 } }
I20260620 18:32:22.216531 13885 raft_consensus.cc:388] T 00000000000000000000000000000000 P af2a6b0d7f6748369de043ce1aadae4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:32:22.216681 13885 raft_consensus.cc:482] T 00000000000000000000000000000000 P af2a6b0d7f6748369de043ce1aadae4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:32:22.216882 13885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af2a6b0d7f6748369de043ce1aadae4e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:22.219049 13885 raft_consensus.cc:504] T 00000000000000000000000000000000 P af2a6b0d7f6748369de043ce1aadae4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2a6b0d7f6748369de043ce1aadae4e" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 39993 } }
I20260620 18:32:22.219523 13885 leader_election.cc:302] T 00000000000000000000000000000000 P af2a6b0d7f6748369de043ce1aadae4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af2a6b0d7f6748369de043ce1aadae4e; no voters:
I20260620 18:32:22.220069 13885 leader_election.cc:288] T 00000000000000000000000000000000 P af2a6b0d7f6748369de043ce1aadae4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:32:22.220193 13890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af2a6b0d7f6748369de043ce1aadae4e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:22.220880 13890 raft_consensus.cc:686] T 00000000000000000000000000000000 P af2a6b0d7f6748369de043ce1aadae4e [term 1 LEADER]: Becoming Leader. State: Replica: af2a6b0d7f6748369de043ce1aadae4e, State: Running, Role: LEADER
I20260620 18:32:22.221390 13890 consensus_queue.cc:237] T 00000000000000000000000000000000 P af2a6b0d7f6748369de043ce1aadae4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2a6b0d7f6748369de043ce1aadae4e" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 39993 } }
I20260620 18:32:22.222149 13885 sys_catalog.cc:564] T 00000000000000000000000000000000 P af2a6b0d7f6748369de043ce1aadae4e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:32:22.224525 13892 sys_catalog.cc:455] T 00000000000000000000000000000000 P af2a6b0d7f6748369de043ce1aadae4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader af2a6b0d7f6748369de043ce1aadae4e. Latest consensus state: current_term: 1 leader_uuid: "af2a6b0d7f6748369de043ce1aadae4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2a6b0d7f6748369de043ce1aadae4e" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 39993 } } }
I20260620 18:32:22.224884 13892 sys_catalog.cc:458] T 00000000000000000000000000000000 P af2a6b0d7f6748369de043ce1aadae4e [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:22.224606 13891 sys_catalog.cc:455] T 00000000000000000000000000000000 P af2a6b0d7f6748369de043ce1aadae4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af2a6b0d7f6748369de043ce1aadae4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2a6b0d7f6748369de043ce1aadae4e" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 39993 } } }
I20260620 18:32:22.225077 13891 sys_catalog.cc:458] T 00000000000000000000000000000000 P af2a6b0d7f6748369de043ce1aadae4e [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:22.226794 13896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:32:22.231973 13896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:32:22.240384 13896 catalog_manager.cc:1347] Generated new cluster ID: faa33b1f17354932b89ffbfd3aac3df0
I20260620 18:32:22.240495 13896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:32:22.253077 13896 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:32:22.254850 13896 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:32:22.269609 13896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af2a6b0d7f6748369de043ce1aadae4e: Generated new TSK 0
I20260620 18:32:22.270720 13896 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.131.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab.
Entry for principal kudu/127.10.131.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab.
I20260620 18:32:22.316298 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.1:0
--local_ip_for_outbound_sockets=127.10.131.1
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:39993
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:42789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:22.638244 13911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:22.638523 13911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:22.638664 13911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:22.645953 13911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:22.646054 13911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:22.646251 13911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.1
I20260620 18:32:22.656293 13911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:42789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:39993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.1
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:22.657656 13911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:22.658969 13911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:22.671744 13916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:22.671825 13919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:22.671880 13917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:22.672518 13911 server_base.cc:1034] running on GCE node
I20260620 18:32:22.673136 13911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:22.674011 13911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:22.675266 13911 hybrid_clock.cc:648] HybridClock initialized: now 1781980342675212 us; error 58 us; skew 500 ppm
I20260620 18:32:22.675603 13911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:22 dist-test-slave-cbn3 krb5kdc[13805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980342, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:22.678748 13911 init.cc:377] Logged in from keytab as kudu/127.10.131.1@KRBTEST.COM (short username kudu)
I20260620 18:32:22.680040 13911 webserver.cc:460] Webserver started at http://127.10.131.1:40547/ using document root <none> and password file <none>
I20260620 18:32:22.680649 13911 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:22.680751 13911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:22.681064 13911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:22.683804 13911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-0/data/instance:
uuid: "e7998634dc884282a9e0b83d5a0fbfaa"
format_stamp: "Formatted at 2026-06-20 18:32:22 on dist-test-slave-cbn3"
server_key: "2b672586fab16323055e5107ba09e22b"
server_key_iv: "b25c802d9cfc7f9298d60e915549862e"
server_key_version: "encryptionkey@0"
I20260620 18:32:22.684504 13911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance:
uuid: "e7998634dc884282a9e0b83d5a0fbfaa"
format_stamp: "Formatted at 2026-06-20 18:32:22 on dist-test-slave-cbn3"
server_key: "2b672586fab16323055e5107ba09e22b"
server_key_iv: "b25c802d9cfc7f9298d60e915549862e"
server_key_version: "encryptionkey@0"
I20260620 18:32:22.689766 13911 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 18:32:22.693073 13926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:22.694316 13911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:32:22.694492 13911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-0/wal
uuid: "e7998634dc884282a9e0b83d5a0fbfaa"
format_stamp: "Formatted at 2026-06-20 18:32:22 on dist-test-slave-cbn3"
server_key: "2b672586fab16323055e5107ba09e22b"
server_key_iv: "b25c802d9cfc7f9298d60e915549862e"
server_key_version: "encryptionkey@0"
I20260620 18:32:22.694671 13911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:22.708815 13911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:22.712814 13911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:22.713176 13911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:22.714085 13911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:22.715481 13911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:22.715552 13911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:22.715631 13911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:22.715677 13911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:22.749292 13911 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.1:41323
I20260620 18:32:22.749379 14038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.1:41323 every 8 connection(s)
I20260620 18:32:22.750548 13911 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
I20260620 18:32:22.755333 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 13911
I20260620 18:32:22.755575 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance
I20260620 18:32:22.756366 10764 external_mini_cluster.cc:1428] Setting key 014d0facd09b49092f747b2d9023c801
Jun 20 18:32:22 dist-test-slave-cbn3 krb5kdc[13805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980342, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:32:22.771999 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:22.753034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:48755 (local address 127.10.131.62:39993)
0620 18:32:22.753450 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:22.753455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:22.754376 (+ 921us) server_negotiation.cc:407] Connection header received
0620 18:32:22.757000 (+ 2624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:22.757008 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:22.757099 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:22.757234 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:22.758984 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:22.759689 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:22.762173 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:22.762557 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:22.762650 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:22.766284 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:22.766289 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:22.766293 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:22.766324 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:22.768804 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:22.769696 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:22.769700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:22.769702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:22.769837 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:22.770285 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:22.770289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:22.770291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:22.770559 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:22.770717 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:22.771413 (+ 696us) server_negotiation.cc:299] Negotiation successful
0620 18:32:22.771627 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":211,"threads_started":1}
I20260620 18:32:22.773077 14042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:22.753571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:39993 (local address 127.10.131.1:48755)
0620 18:32:22.754228 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:22.754266 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:22.756596 (+ 2330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:22.757524 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:22.757533 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:22.758008 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:22.758796 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:22.758822 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:22.759847 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:22.759855 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:22.762016 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:22.762034 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:22.762784 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:22.762793 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:22.763118 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:22.763858 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:22.763890 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:22.766054 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:22.768977 (+ 2923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:22.768986 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:22.769003 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:22.769544 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:22.769952 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:22.769957 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:22.769961 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:22.770153 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:22.770690 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:22.770697 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:22.771058 (+ 361us) client_negotiation.cc:769] Sending connection context
0620 18:32:22.771322 (+ 264us) client_negotiation.cc:240] Negotiation successful
0620 18:32:22.771863 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":258,"threads_started":1}
I20260620 18:32:22.774664 14039 heartbeater.cc:347] Connected to a master server at 127.10.131.62:39993
WARNING: no policy specified for kudu/127.10.131.2@KRBTEST.COM; defaulting to no policy
I20260620 18:32:22.774967 14039 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:22.775684 14039 heartbeater.cc:511] Master 127.10.131.62:39993 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.2@KRBTEST.COM" created.
I20260620 18:32:22.777977 13850 ts_manager.cc:194] Registered new tserver with Master: e7998634dc884282a9e0b83d5a0fbfaa (127.10.131.1:41323)
I20260620 18:32:22.779474 13850 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.1@KRBTEST.COM'} at 127.10.131.1:48755
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab.
Entry for principal kudu/127.10.131.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab.
I20260620 18:32:22.799270 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.2:0
--local_ip_for_outbound_sockets=127.10.131.2
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:39993
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:42789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:23.109757 14045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:23.110020 14045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:23.110111 14045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:23.117172 14045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:23.117267 14045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:23.117378 14045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.2
I20260620 18:32:23.126745 14045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:42789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:39993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.2
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:23.128003 14045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:23.129379 14045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:23.143409 14051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:23.143527 14053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:23.143429 14050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:23.144127 14045 server_base.cc:1034] running on GCE node
I20260620 18:32:23.144733 14045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:23.145680 14045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:23.146927 14045 hybrid_clock.cc:648] HybridClock initialized: now 1781980343146874 us; error 46 us; skew 500 ppm
I20260620 18:32:23.147243 14045 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:23 dist-test-slave-cbn3 krb5kdc[13805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980343, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:23.150453 14045 init.cc:377] Logged in from keytab as kudu/127.10.131.2@KRBTEST.COM (short username kudu)
I20260620 18:32:23.151963 14045 webserver.cc:460] Webserver started at http://127.10.131.2:44081/ using document root <none> and password file <none>
I20260620 18:32:23.152654 14045 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:23.152801 14045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:23.153126 14045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:23.156066 14045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-1/data/instance:
uuid: "7cf9ca5509114efb8db299ac43198dd0"
format_stamp: "Formatted at 2026-06-20 18:32:23 on dist-test-slave-cbn3"
server_key: "e9412c241c309025e9db41c84875a2a5"
server_key_iv: "105c9d8093b52962cf5cdf2a5ce9d519"
server_key_version: "encryptionkey@0"
I20260620 18:32:23.156824 14045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance:
uuid: "7cf9ca5509114efb8db299ac43198dd0"
format_stamp: "Formatted at 2026-06-20 18:32:23 on dist-test-slave-cbn3"
server_key: "e9412c241c309025e9db41c84875a2a5"
server_key_iv: "105c9d8093b52962cf5cdf2a5ce9d519"
server_key_version: "encryptionkey@0"
I20260620 18:32:23.161873 14045 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260620 18:32:23.165222 14060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:23.166544 14045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 18:32:23.166734 14045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-1/wal
uuid: "7cf9ca5509114efb8db299ac43198dd0"
format_stamp: "Formatted at 2026-06-20 18:32:23 on dist-test-slave-cbn3"
server_key: "e9412c241c309025e9db41c84875a2a5"
server_key_iv: "105c9d8093b52962cf5cdf2a5ce9d519"
server_key_version: "encryptionkey@0"
I20260620 18:32:23.166942 14045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:23.183463 14045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:23.187661 14045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:23.187968 14045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:23.188989 14045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:23.190310 14045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:23.190411 14045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:23.190559 14045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:23.190630 14045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:23.224167 14045 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.2:34541
I20260620 18:32:23.224253 14172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.2:34541 every 8 connection(s)
I20260620 18:32:23.225459 14045 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
I20260620 18:32:23.225894 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 14045
I20260620 18:32:23.226094 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance
I20260620 18:32:23.226717 10764 external_mini_cluster.cc:1428] Setting key c36b060e361aba0fc3f16be2625f888f
Jun 20 18:32:23 dist-test-slave-cbn3 krb5kdc[13805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980343, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.131.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.3@KRBTEST.COM" created.
I20260620 18:32:23.249778 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:23.228484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:53081 (local address 127.10.131.62:39993)
0620 18:32:23.228655 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:23.228660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:23.230067 (+ 1407us) server_negotiation.cc:407] Connection header received
0620 18:32:23.232905 (+ 2838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:23.232913 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:23.232987 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:23.233072 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:23.235093 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:23.236395 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:23.238040 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:23.238803 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:23.238944 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:23.242484 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:23.242493 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:23.242501 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:23.242561 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:23.246362 (+ 3801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:23.247368 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:23.247377 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:23.247381 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:23.247536 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:23.247999 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:23.248003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:23.248005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:23.248311 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:23.248444 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:23.249259 (+ 815us) server_negotiation.cc:299] Negotiation successful
0620 18:32:23.249450 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 18:32:23.250687 14175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:23.229125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:39993 (local address 127.10.131.2:53081)
0620 18:32:23.229911 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:23.229955 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:23.232559 (+ 2604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:23.233328 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:23.233342 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:23.234086 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:23.234863 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:23.234891 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:23.236544 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:23.236548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:23.237904 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:23.237916 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:23.239005 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:23.239009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:23.239196 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:23.239872 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:23.239894 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:23.242251 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:23.246552 (+ 4301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:23.246563 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:23.246583 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:23.247197 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:23.247654 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:23.247657 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:23.247659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:23.247857 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:23.248438 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:23.248446 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:23.248766 (+ 320us) client_negotiation.cc:769] Sending connection context
0620 18:32:23.249113 (+ 347us) client_negotiation.cc:240] Negotiation successful
0620 18:32:23.249510 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":291,"threads_started":1}
I20260620 18:32:23.252197 14173 heartbeater.cc:347] Connected to a master server at 127.10.131.62:39993
I20260620 18:32:23.252501 14173 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:23.253293 14173 heartbeater.cc:511] Master 127.10.131.62:39993 requested a full tablet report, sending...
I20260620 18:32:23.254922 13850 ts_manager.cc:194] Registered new tserver with Master: 7cf9ca5509114efb8db299ac43198dd0 (127.10.131.2:34541)
I20260620 18:32:23.256006 13850 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.2@KRBTEST.COM'} at 127.10.131.2:53081
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab.
Entry for principal kudu/127.10.131.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab.
I20260620 18:32:23.272370 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.3:0
--local_ip_for_outbound_sockets=127.10.131.3
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:39993
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:42789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:23.586390 14178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:23.586650 14178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:23.586745 14178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:23.593748 14178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:23.593856 14178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:23.594089 14178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.3
I20260620 18:32:23.603755 14178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:42789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:39993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.3
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:23.605050 14178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:23.606427 14178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:23.621790 14184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:23.621833 14183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:23.622612 14186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:23.623826 14178 server_base.cc:1034] running on GCE node
I20260620 18:32:23.624421 14178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:23.625386 14178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:23.626606 14178 hybrid_clock.cc:648] HybridClock initialized: now 1781980343626551 us; error 37 us; skew 500 ppm
I20260620 18:32:23.626969 14178 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:23 dist-test-slave-cbn3 krb5kdc[13805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980343, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:23.630093 14178 init.cc:377] Logged in from keytab as kudu/127.10.131.3@KRBTEST.COM (short username kudu)
I20260620 18:32:23.631631 14178 webserver.cc:460] Webserver started at http://127.10.131.3:38171/ using document root <none> and password file <none>
I20260620 18:32:23.632357 14178 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:23.632499 14178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:23.632797 14178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:23.635973 14178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-2/data/instance:
uuid: "5d056145d533482aa8772815319acbee"
format_stamp: "Formatted at 2026-06-20 18:32:23 on dist-test-slave-cbn3"
server_key: "e926c23ca20943dafaec5b6f7789e34c"
server_key_iv: "c2a378b7f781b4105f7d3747c700f77b"
server_key_version: "encryptionkey@0"
I20260620 18:32:23.636785 14178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance:
uuid: "5d056145d533482aa8772815319acbee"
format_stamp: "Formatted at 2026-06-20 18:32:23 on dist-test-slave-cbn3"
server_key: "e926c23ca20943dafaec5b6f7789e34c"
server_key_iv: "c2a378b7f781b4105f7d3747c700f77b"
server_key_version: "encryptionkey@0"
I20260620 18:32:23.642025 14178 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:32:23.645505 14193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:23.646855 14178 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:32:23.647050 14178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-2/wal
uuid: "5d056145d533482aa8772815319acbee"
format_stamp: "Formatted at 2026-06-20 18:32:23 on dist-test-slave-cbn3"
server_key: "e926c23ca20943dafaec5b6f7789e34c"
server_key_iv: "c2a378b7f781b4105f7d3747c700f77b"
server_key_version: "encryptionkey@0"
I20260620 18:32:23.647257 14178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:23.661393 14178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:23.665503 14178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:23.665808 14178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:23.666867 14178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:23.668342 14178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:23.668447 14178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:23.668597 14178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:23.668670 14178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:23.702622 14178 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.3:45255
I20260620 18:32:23.702715 14305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.3:45255 every 8 connection(s)
I20260620 18:32:23.703866 14178 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
I20260620 18:32:23.709190 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 14178
I20260620 18:32:23.709380 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance
I20260620 18:32:23.709975 10764 external_mini_cluster.cc:1428] Setting key c30ce816882369f0d0c671455da3c966
Jun 20 18:32:23 dist-test-slave-cbn3 krb5kdc[13805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980343, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:32:23.724262 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:23.706474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:37571 (local address 127.10.131.62:39993)
0620 18:32:23.706664 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:23.706668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:23.707777 (+ 1109us) server_negotiation.cc:407] Connection header received
0620 18:32:23.710117 (+ 2340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:23.710125 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:23.710227 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:23.710334 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:23.712398 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:23.713455 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:23.714957 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:23.715349 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:23.715445 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:23.718830 (+ 3385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:23.718835 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:23.718838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:23.718868 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:23.721337 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:23.722043 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:23.722047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:23.722049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:23.722161 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:23.722594 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:23.722598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:23.722600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:23.722863 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:23.722977 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:23.723655 (+ 678us) server_negotiation.cc:299] Negotiation successful
0620 18:32:23.723919 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:32:23.725414 14308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:23.706941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:39993 (local address 127.10.131.3:37571)
0620 18:32:23.707616 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:23.707658 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:23.709767 (+ 2109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:23.710596 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:23.710604 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:23.711386 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:23.712213 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:23.712244 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:23.713587 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:23.713591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:23.714832 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:23.714843 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:23.715562 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:23.715567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:23.715774 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:23.716463 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:23.716484 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:23.718645 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:23.721462 (+ 2817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:23.721468 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:23.721480 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:23.721874 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:23.722262 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:23.722266 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:23.722268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:23.722507 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:23.722966 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:23.722972 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:23.723289 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 18:32:23.723576 (+ 287us) client_negotiation.cc:240] Negotiation successful
0620 18:32:23.724062 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":262,"threads_started":1}
I20260620 18:32:23.726713 14306 heartbeater.cc:347] Connected to a master server at 127.10.131.62:39993
I20260620 18:32:23.726994 14306 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:23.727732 14306 heartbeater.cc:511] Master 127.10.131.62:39993 requested a full tablet report, sending...
I20260620 18:32:23.729141 13850 ts_manager.cc:194] Registered new tserver with Master: 5d056145d533482aa8772815319acbee (127.10.131.3:45255)
I20260620 18:32:23.729867 13850 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.3@KRBTEST.COM'} at 127.10.131.3:37571
I20260620 18:32:23.737792 10764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 18:32:23 dist-test-slave-cbn3 krb5kdc[13805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980343, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 18:32:23 dist-test-slave-cbn3 krb5kdc[13805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980343, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:32:23.782359 14039 heartbeater.cc:503] Master 127.10.131.62:39993 was elected leader, sending a full tablet report...
I20260620 18:32:23.786454 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:23.772415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46712 (local address 127.10.131.62:39993)
0620 18:32:23.772600 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:23.772605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:23.773203 (+ 598us) server_negotiation.cc:407] Connection header received
0620 18:32:23.773339 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:23.773344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:23.773414 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:23.773510 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:23.774843 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:23.775582 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:23.776759 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:23.777230 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:23.777325 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:23.781125 (+ 3800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:23.781129 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:23.781133 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:23.781166 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:23.784012 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:23.784626 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:23.784630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:23.784633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:23.784759 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:23.785152 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:23.785156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:23.785158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:23.785475 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:23.785655 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:23.785969 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 18:32:23.786123 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 18:32:23.786980 13850 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46712
I20260620 18:32:23.788578 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 13911
Jun 20 18:32:23 dist-test-slave-cbn3 krb5kdc[13805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980343, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.10.131.1@KRBTEST.COM
I20260620 18:32:23.803393 14317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:23.789563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39342 (local address 127.10.131.1:41323)
0620 18:32:23.790128 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:23.790135 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:23.790157 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:32:23.790204 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:23.790210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:23.790390 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:23.790514 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:23.791584 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:23.792632 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:23.793890 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:23.794428 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:23.794534 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:23.798246 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:23.798256 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:23.798269 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:23.798301 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:23.801065 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:23.801702 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:23.801709 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:23.801711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:23.801825 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:23.802163 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:23.802166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:23.802168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:23.802468 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:23.802678 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:23.802930 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 18:32:23.803120 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":283,"threads_started":1}
W20260620 18:32:23.804523 14034 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39342
W20260620 18:32:23.805593 10764 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 13911: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:32:23.805765 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 13911
I20260620 18:32:23.821347 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 14045
Jun 20 18:32:23 dist-test-slave-cbn3 krb5kdc[13805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980343, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.10.131.2@KRBTEST.COM
I20260620 18:32:23.836966 14318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:23.822080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38246 (local address 127.10.131.2:34541)
0620 18:32:23.822697 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:23.822704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:23.822728 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:32:23.822806 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:23.822812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:23.823060 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:23.823229 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:23.824407 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:23.825456 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:23.826673 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:23.827192 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:23.827330 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:23.831479 (+ 4149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:23.831488 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:23.831500 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:23.831532 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:23.834271 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:23.834848 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:23.834854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:23.834856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:23.835001 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:23.835314 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:23.835317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:23.835320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:23.835738 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:23.835985 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:23.836265 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 18:32:23.836563 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":303,"threads_started":1}
W20260620 18:32:23.838165 14168 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:38246
W20260620 18:32:23.839181 10764 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 14045: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:32:23.839322 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 14045
I20260620 18:32:23.854910 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 14178
Jun 20 18:32:23 dist-test-slave-cbn3 krb5kdc[13805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980343, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.10.131.3@KRBTEST.COM
I20260620 18:32:23.871788 14319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:23.855875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36334 (local address 127.10.131.3:45255)
0620 18:32:23.856490 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:23.856496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:23.856524 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:32:23.856581 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:23.856588 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:23.856790 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:23.857057 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:23.858472 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:23.859634 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:23.860938 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:23.861505 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:23.861647 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:23.866165 (+ 4518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:23.866177 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:23.866192 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:23.866227 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:23.869165 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:23.869896 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:23.869903 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:23.869905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:23.870027 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:23.870429 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:23.870432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:23.870434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:23.870774 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:23.871004 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:23.871300 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 18:32:23.871513 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":323,"threads_started":1}
W20260620 18:32:23.873359 14301 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36334
W20260620 18:32:23.874352 10764 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 14178: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:32:23.874509 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 14178
I20260620 18:32:23.889822 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 13818
I20260620 18:32:23.891124 13880 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:23.991739 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 13818
2026-06-20T18:32:24Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3513 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:32:24 dist-test-slave-cbn3 krb5kdc[14323](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:32:24 dist-test-slave-cbn3 krb5kdc[14323](info): set up 2 sockets
Jun 20 18:32:24 dist-test-slave-cbn3 krb5kdc[14323](info): commencing operation
krb5kdc: starting...
W20260620 18:32:25.073617 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:32:25 dist-test-slave-cbn3 krb5kdc[14323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:32:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:32:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.131.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
I20260620 18:32:25.260744 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:44529
--webserver_interface=127.10.131.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:45613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:44529
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:25.574905 14336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:25.575225 14336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:25.575323 14336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:25.582564 14336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:32:25.582670 14336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:25.582720 14336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:25.582782 14336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:32:25.582831 14336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:32:25.592736 14336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:45613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:44529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:44529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:25.594034 14336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:25.595348 14336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:25.606649 14341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:25.606693 14342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:25.606729 14344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:25.607900 14336 server_base.cc:1034] running on GCE node
I20260620 18:32:25.608979 14336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:25.610625 14336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:25.611907 14336 hybrid_clock.cc:648] HybridClock initialized: now 1781980345611800 us; error 88 us; skew 500 ppm
I20260620 18:32:25.612377 14336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:25 dist-test-slave-cbn3 krb5kdc[14323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980345, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:25.619143 14336 init.cc:377] Logged in from keytab as kudu/127.10.131.62@KRBTEST.COM (short username kudu)
I20260620 18:32:25.620687 14336 webserver.cc:460] Webserver started at http://127.10.131.62:37741/ using document root <none> and password file <none>
I20260620 18:32:25.621417 14336 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:25.621551 14336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:25.621855 14336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:25.624720 14336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "c917b35d29fe49ac91c8c2ddc7e2d06d"
format_stamp: "Formatted at 2026-06-20 18:32:25 on dist-test-slave-cbn3"
server_key: "e0de3acb60181c6d1847304da3bac26e"
server_key_iv: "eace7f9900b74efa3c1784e720faebee"
server_key_version: "encryptionkey@0"
I20260620 18:32:25.625545 14336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "c917b35d29fe49ac91c8c2ddc7e2d06d"
format_stamp: "Formatted at 2026-06-20 18:32:25 on dist-test-slave-cbn3"
server_key: "e0de3acb60181c6d1847304da3bac26e"
server_key_iv: "eace7f9900b74efa3c1784e720faebee"
server_key_version: "encryptionkey@0"
I20260620 18:32:25.630478 14336 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:32:25.634104 14351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:25.635421 14336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:32:25.635610 14336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "c917b35d29fe49ac91c8c2ddc7e2d06d"
format_stamp: "Formatted at 2026-06-20 18:32:25 on dist-test-slave-cbn3"
server_key: "e0de3acb60181c6d1847304da3bac26e"
server_key_iv: "eace7f9900b74efa3c1784e720faebee"
server_key_version: "encryptionkey@0"
I20260620 18:32:25.635815 14336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:25.657152 14336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:25.660988 14336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:25.661263 14336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:25.679373 14336 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:44529
I20260620 18:32:25.679391 14402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:44529 every 8 connection(s)
I20260620 18:32:25.680662 14336 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:32:25.684284 14403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:25.687731 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 14336
I20260620 18:32:25.687956 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:32:25.688721 10764 external_mini_cluster.cc:1428] Setting key caf410e14a323647326d1a678990e844
I20260620 18:32:25.696331 14403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c917b35d29fe49ac91c8c2ddc7e2d06d: Bootstrap starting.
I20260620 18:32:25.699286 14403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c917b35d29fe49ac91c8c2ddc7e2d06d: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:25.700276 14403 log.cc:824] T 00000000000000000000000000000000 P c917b35d29fe49ac91c8c2ddc7e2d06d: Log is configured to *not* fsync() on all Append() calls
Jun 20 18:32:25 dist-test-slave-cbn3 krb5kdc[14323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:32:25.703367 14403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c917b35d29fe49ac91c8c2ddc7e2d06d: No bootstrap required, opened a new log
I20260620 18:32:25.709925 14406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:25.690781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57258 (local address 127.10.131.62:44529)
0620 18:32:25.691471 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:25.691482 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:25.691518 (+ 36us) server_negotiation.cc:407] Connection header received
0620 18:32:25.693183 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:25.693204 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:25.693576 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:25.693967 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:25.695377 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:25.696703 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:25.698009 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:25.698648 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:25.698838 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:25.702772 (+ 3934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:25.702779 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:25.702792 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:25.702825 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:25.705943 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:25.706610 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:25.706628 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:25.706634 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:25.706802 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:25.707180 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:25.707187 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:25.707190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:25.707711 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:25.708010 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:25.708453 (+ 443us) server_negotiation.cc:299] Negotiation successful
0620 18:32:25.708930 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":323,"threads_started":1}
I20260620 18:32:25.711148 14403 raft_consensus.cc:348] T 00000000000000000000000000000000 P c917b35d29fe49ac91c8c2ddc7e2d06d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c917b35d29fe49ac91c8c2ddc7e2d06d" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44529 } }
I20260620 18:32:25.711526 14403 raft_consensus.cc:374] T 00000000000000000000000000000000 P c917b35d29fe49ac91c8c2ddc7e2d06d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:25.711629 14403 raft_consensus.cc:729] T 00000000000000000000000000000000 P c917b35d29fe49ac91c8c2ddc7e2d06d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c917b35d29fe49ac91c8c2ddc7e2d06d, State: Initialized, Role: FOLLOWER
I20260620 18:32:25.712245 14403 consensus_queue.cc:260] T 00000000000000000000000000000000 P c917b35d29fe49ac91c8c2ddc7e2d06d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c917b35d29fe49ac91c8c2ddc7e2d06d" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44529 } }
I20260620 18:32:25.712517 14403 raft_consensus.cc:388] T 00000000000000000000000000000000 P c917b35d29fe49ac91c8c2ddc7e2d06d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:32:25.712659 14403 raft_consensus.cc:482] T 00000000000000000000000000000000 P c917b35d29fe49ac91c8c2ddc7e2d06d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:32:25.712832 14403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c917b35d29fe49ac91c8c2ddc7e2d06d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:25.714962 14403 raft_consensus.cc:504] T 00000000000000000000000000000000 P c917b35d29fe49ac91c8c2ddc7e2d06d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c917b35d29fe49ac91c8c2ddc7e2d06d" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44529 } }
I20260620 18:32:25.715546 14403 leader_election.cc:302] T 00000000000000000000000000000000 P c917b35d29fe49ac91c8c2ddc7e2d06d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c917b35d29fe49ac91c8c2ddc7e2d06d; no voters:
I20260620 18:32:25.716485 14403 leader_election.cc:288] T 00000000000000000000000000000000 P c917b35d29fe49ac91c8c2ddc7e2d06d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:32:25.716578 14408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c917b35d29fe49ac91c8c2ddc7e2d06d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:25.717289 14408 raft_consensus.cc:686] T 00000000000000000000000000000000 P c917b35d29fe49ac91c8c2ddc7e2d06d [term 1 LEADER]: Becoming Leader. State: Replica: c917b35d29fe49ac91c8c2ddc7e2d06d, State: Running, Role: LEADER
I20260620 18:32:25.717929 14408 consensus_queue.cc:237] T 00000000000000000000000000000000 P c917b35d29fe49ac91c8c2ddc7e2d06d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c917b35d29fe49ac91c8c2ddc7e2d06d" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44529 } }
I20260620 18:32:25.718735 14403 sys_catalog.cc:564] T 00000000000000000000000000000000 P c917b35d29fe49ac91c8c2ddc7e2d06d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:32:25.720427 14410 sys_catalog.cc:455] T 00000000000000000000000000000000 P c917b35d29fe49ac91c8c2ddc7e2d06d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c917b35d29fe49ac91c8c2ddc7e2d06d. Latest consensus state: current_term: 1 leader_uuid: "c917b35d29fe49ac91c8c2ddc7e2d06d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c917b35d29fe49ac91c8c2ddc7e2d06d" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44529 } } }
I20260620 18:32:25.720858 14410 sys_catalog.cc:458] T 00000000000000000000000000000000 P c917b35d29fe49ac91c8c2ddc7e2d06d [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:25.721930 14409 sys_catalog.cc:455] T 00000000000000000000000000000000 P c917b35d29fe49ac91c8c2ddc7e2d06d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c917b35d29fe49ac91c8c2ddc7e2d06d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c917b35d29fe49ac91c8c2ddc7e2d06d" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44529 } } }
I20260620 18:32:25.722221 14409 sys_catalog.cc:458] T 00000000000000000000000000000000 P c917b35d29fe49ac91c8c2ddc7e2d06d [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:25.722700 14416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:32:25.728368 14416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:32:25.737223 14416 catalog_manager.cc:1347] Generated new cluster ID: ac5baadf00c447a6a7c57aa26e935340
I20260620 18:32:25.737335 14416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:32:25.747833 14416 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:32:25.749305 14416 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:32:25.754681 14416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c917b35d29fe49ac91c8c2ddc7e2d06d: Generated new TSK 0
I20260620 18:32:25.755641 14416 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.131.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab.
Entry for principal kudu/127.10.131.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab.
I20260620 18:32:25.798899 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.1:0
--local_ip_for_outbound_sockets=127.10.131.1
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:44529
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:45613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:26.114382 14429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:26.114651 14429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:26.114822 14429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:26.122413 14429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:26.122519 14429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:26.122692 14429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.1
I20260620 18:32:26.132575 14429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:45613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:44529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.1
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:26.133905 14429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:26.135239 14429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:26.148581 14434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:26.148674 14435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:26.148631 14437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:26.149789 14429 server_base.cc:1034] running on GCE node
I20260620 18:32:26.150322 14429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:26.151183 14429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:26.152379 14429 hybrid_clock.cc:648] HybridClock initialized: now 1781980346152320 us; error 53 us; skew 500 ppm
I20260620 18:32:26.152698 14429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:26 dist-test-slave-cbn3 krb5kdc[14323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980346, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:26.155685 14429 init.cc:377] Logged in from keytab as kudu/127.10.131.1@KRBTEST.COM (short username kudu)
I20260620 18:32:26.157155 14429 webserver.cc:460] Webserver started at http://127.10.131.1:45361/ using document root <none> and password file <none>
I20260620 18:32:26.157810 14429 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:26.157934 14429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:26.158241 14429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:26.161147 14429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-0/data/instance:
uuid: "2803e6876793492fa7e5da00ab475b94"
format_stamp: "Formatted at 2026-06-20 18:32:26 on dist-test-slave-cbn3"
server_key: "2c121abcd2738e8b828976890ce399ce"
server_key_iv: "e46c27af28c2e74fbaaf222a3dec5c9c"
server_key_version: "encryptionkey@0"
I20260620 18:32:26.161934 14429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance:
uuid: "2803e6876793492fa7e5da00ab475b94"
format_stamp: "Formatted at 2026-06-20 18:32:26 on dist-test-slave-cbn3"
server_key: "2c121abcd2738e8b828976890ce399ce"
server_key_iv: "e46c27af28c2e74fbaaf222a3dec5c9c"
server_key_version: "encryptionkey@0"
I20260620 18:32:26.167063 14429 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:32:26.170393 14444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:26.171617 14429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:32:26.171788 14429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-0/wal
uuid: "2803e6876793492fa7e5da00ab475b94"
format_stamp: "Formatted at 2026-06-20 18:32:26 on dist-test-slave-cbn3"
server_key: "2c121abcd2738e8b828976890ce399ce"
server_key_iv: "e46c27af28c2e74fbaaf222a3dec5c9c"
server_key_version: "encryptionkey@0"
I20260620 18:32:26.171978 14429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:26.185900 14429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:26.189875 14429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:26.190162 14429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:26.191073 14429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:26.192306 14429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:26.192406 14429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:26.192550 14429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:26.192631 14429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:26.226045 14429 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.1:45075
I20260620 18:32:26.226092 14556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.1:45075 every 8 connection(s)
I20260620 18:32:26.227394 14429 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
I20260620 18:32:26.228801 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 14429
I20260620 18:32:26.229061 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance
I20260620 18:32:26.231120 10764 external_mini_cluster.cc:1428] Setting key 06383096f859a4a1a8a35ca326c9b3e4
Jun 20 18:32:26 dist-test-slave-cbn3 krb5kdc[14323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980346, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.131.2@KRBTEST.COM; defaulting to no policy
I20260620 18:32:26.250582 14559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:26.230348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:60613 (local address 127.10.131.62:44529)
0620 18:32:26.233197 (+ 2849us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:26.233205 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:26.233233 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:32:26.234679 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:26.234689 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:26.234779 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:26.234916 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:26.236653 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:26.237828 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:26.240160 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:26.240676 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:26.240774 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:26.244967 (+ 4193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:26.244971 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:26.244974 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:26.245004 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:26.247612 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:26.248325 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:26.248331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:26.248340 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:26.248450 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:26.248905 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:26.248909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:26.248912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:26.249177 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:26.249334 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:26.250007 (+ 673us) server_negotiation.cc:299] Negotiation successful
0620 18:32:26.250214 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2704,"thread_start_us":3138,"threads_started":1}
I20260620 18:32:26.251524 14560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:26.230766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44529 (local address 127.10.131.1:60613)
0620 18:32:26.232428 (+ 1662us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:26.232475 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:26.234409 (+ 1934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:26.235232 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:26.235241 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:26.235829 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:26.236454 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:26.236474 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:26.238017 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:26.238021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:26.240014 (+ 1993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:26.240027 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:26.240929 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:26.240939 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:26.241282 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:26.242014 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:26.242049 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:26.244713 (+ 2664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:26.247764 (+ 3051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:26.247773 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:26.247797 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:26.248198 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:26.248563 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:26.248568 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:26.248572 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:26.248757 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:26.249302 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:26.249310 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:26.249658 (+ 348us) client_negotiation.cc:769] Sending connection context
0620 18:32:26.249914 (+ 256us) client_negotiation.cc:240] Negotiation successful
0620 18:32:26.250423 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1355,"thread_start_us":1141,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.2@KRBTEST.COM" created.
I20260620 18:32:26.253129 14557 heartbeater.cc:347] Connected to a master server at 127.10.131.62:44529
I20260620 18:32:26.253446 14557 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:26.254317 14557 heartbeater.cc:511] Master 127.10.131.62:44529 requested a full tablet report, sending...
I20260620 18:32:26.256448 14368 ts_manager.cc:194] Registered new tserver with Master: 2803e6876793492fa7e5da00ab475b94 (127.10.131.1:45075)
I20260620 18:32:26.258394 14368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.1@KRBTEST.COM'} at 127.10.131.1:60613
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab.
Entry for principal kudu/127.10.131.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab.
I20260620 18:32:26.276185 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.2:0
--local_ip_for_outbound_sockets=127.10.131.2
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:44529
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:45613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:26.600571 14563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:26.600888 14563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:26.601047 14563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:26.608440 14563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:26.608548 14563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:26.608774 14563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.2
I20260620 18:32:26.618655 14563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:45613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:44529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.2
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:26.620105 14563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:26.621477 14563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:26.634872 14569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:26.634828 14571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:26.634918 14568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:26.636013 14563 server_base.cc:1034] running on GCE node
I20260620 18:32:26.636641 14563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:26.637715 14563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:26.638937 14563 hybrid_clock.cc:648] HybridClock initialized: now 1781980346638882 us; error 49 us; skew 500 ppm
I20260620 18:32:26.639271 14563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:26 dist-test-slave-cbn3 krb5kdc[14323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980346, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:26.642521 14563 init.cc:377] Logged in from keytab as kudu/127.10.131.2@KRBTEST.COM (short username kudu)
I20260620 18:32:26.643996 14563 webserver.cc:460] Webserver started at http://127.10.131.2:33183/ using document root <none> and password file <none>
I20260620 18:32:26.644637 14563 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:26.644737 14563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:26.645081 14563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:26.647956 14563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-1/data/instance:
uuid: "ff192094b8264f089ed84a6e41af4ebb"
format_stamp: "Formatted at 2026-06-20 18:32:26 on dist-test-slave-cbn3"
server_key: "10dd3a226748d2105490359b5f970cbd"
server_key_iv: "57760f9683c998428e36a994d943c9eb"
server_key_version: "encryptionkey@0"
I20260620 18:32:26.648689 14563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance:
uuid: "ff192094b8264f089ed84a6e41af4ebb"
format_stamp: "Formatted at 2026-06-20 18:32:26 on dist-test-slave-cbn3"
server_key: "10dd3a226748d2105490359b5f970cbd"
server_key_iv: "57760f9683c998428e36a994d943c9eb"
server_key_version: "encryptionkey@0"
I20260620 18:32:26.653890 14563 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:32:26.657395 14578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:26.658771 14563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 18:32:26.658905 14563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-1/wal
uuid: "ff192094b8264f089ed84a6e41af4ebb"
format_stamp: "Formatted at 2026-06-20 18:32:26 on dist-test-slave-cbn3"
server_key: "10dd3a226748d2105490359b5f970cbd"
server_key_iv: "57760f9683c998428e36a994d943c9eb"
server_key_version: "encryptionkey@0"
I20260620 18:32:26.659097 14563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:26.677660 14563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:26.682025 14563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:26.682319 14563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:26.683256 14563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:26.684576 14563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:26.684645 14563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:26.684809 14563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:26.684897 14563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260620 18:32:26.717919 14563 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.2:39243
I20260620 18:32:26.717968 14690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.2:39243 every 8 connection(s)
I20260620 18:32:26.719241 14563 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
I20260620 18:32:26.724684 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 14563
I20260620 18:32:26.724947 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance
I20260620 18:32:26.725432 10764 external_mini_cluster.cc:1428] Setting key 3af710084d62f83a7eba1fb175bd2697
Jun 20 18:32:26 dist-test-slave-cbn3 krb5kdc[14323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980346, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:32:26.741775 14559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:26.721995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:43161 (local address 127.10.131.62:44529)
0620 18:32:26.722183 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:26.722188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:26.723369 (+ 1181us) server_negotiation.cc:407] Connection header received
0620 18:32:26.725294 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:26.725300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:26.725387 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:26.725516 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:26.727186 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:26.728043 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:26.730595 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:26.731014 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:26.731173 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:26.735492 (+ 4319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:26.735497 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:26.735500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:26.735531 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:26.738183 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:26.739204 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:26.739209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:26.739211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:26.739339 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:26.739899 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:26.739902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:26.739904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:26.740178 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:26.740347 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:26.741278 (+ 931us) server_negotiation.cc:299] Negotiation successful
0620 18:32:26.741502 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 18:32:26.743562 14693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:26.722530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44529 (local address 127.10.131.2:43161)
0620 18:32:26.723222 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:26.723257 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:26.725034 (+ 1777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:26.725750 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:26.725759 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:26.726316 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:26.727015 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:26.727038 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:26.728211 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:26.728218 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:26.730441 (+ 2223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:26.730458 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:26.731307 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:26.731316 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:26.731561 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:26.732294 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:26.732324 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:26.735244 (+ 2920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:26.738334 (+ 3090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:26.738346 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:26.738366 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:26.739049 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:26.739468 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:26.739476 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:26.739483 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:26.739753 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:26.740325 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:26.740336 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:26.740864 (+ 528us) client_negotiation.cc:769] Sending connection context
0620 18:32:26.741206 (+ 342us) client_negotiation.cc:240] Negotiation successful
0620 18:32:26.741818 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":280,"threads_started":1}
I20260620 18:32:26.745221 14691 heartbeater.cc:347] Connected to a master server at 127.10.131.62:44529
I20260620 18:32:26.745545 14691 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:26.746251 14691 heartbeater.cc:511] Master 127.10.131.62:44529 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.131.3@KRBTEST.COM; defaulting to no policy
I20260620 18:32:26.747663 14368 ts_manager.cc:194] Registered new tserver with Master: ff192094b8264f089ed84a6e41af4ebb (127.10.131.2:39243)
I20260620 18:32:26.748428 14368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.2@KRBTEST.COM'} at 127.10.131.2:43161
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab.
Entry for principal kudu/127.10.131.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab.
I20260620 18:32:26.772161 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.3:0
--local_ip_for_outbound_sockets=127.10.131.3
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:44529
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:45613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:27.083693 14696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:27.083950 14696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:27.084129 14696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:27.091287 14696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:27.091400 14696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:27.091612 14696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.3
I20260620 18:32:27.101219 14696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:45613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:44529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.3
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:27.102433 14696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:27.103693 14696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:27.116485 14704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:27.116586 14702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:27.117049 14701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:27.117246 14696 server_base.cc:1034] running on GCE node
I20260620 18:32:27.117892 14696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:27.118889 14696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:27.120100 14696 hybrid_clock.cc:648] HybridClock initialized: now 1781980347120050 us; error 43 us; skew 500 ppm
I20260620 18:32:27.120440 14696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:27 dist-test-slave-cbn3 krb5kdc[14323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980347, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:27.123580 14696 init.cc:377] Logged in from keytab as kudu/127.10.131.3@KRBTEST.COM (short username kudu)
I20260620 18:32:27.125080 14696 webserver.cc:460] Webserver started at http://127.10.131.3:45365/ using document root <none> and password file <none>
I20260620 18:32:27.125805 14696 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:27.125953 14696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:27.126235 14696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:27.129246 14696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-2/data/instance:
uuid: "838e5aa38ebe4020b0fe36223a80fec1"
format_stamp: "Formatted at 2026-06-20 18:32:27 on dist-test-slave-cbn3"
server_key: "373189982aacaafd3eede1cfb3f42ca2"
server_key_iv: "2b3581e59bd04d20cf5f30cbb87af5f7"
server_key_version: "encryptionkey@0"
I20260620 18:32:27.129956 14696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance:
uuid: "838e5aa38ebe4020b0fe36223a80fec1"
format_stamp: "Formatted at 2026-06-20 18:32:27 on dist-test-slave-cbn3"
server_key: "373189982aacaafd3eede1cfb3f42ca2"
server_key_iv: "2b3581e59bd04d20cf5f30cbb87af5f7"
server_key_version: "encryptionkey@0"
I20260620 18:32:27.134958 14696 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 18:32:27.138283 14711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:27.139758 14696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:32:27.139937 14696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-2/wal
uuid: "838e5aa38ebe4020b0fe36223a80fec1"
format_stamp: "Formatted at 2026-06-20 18:32:27 on dist-test-slave-cbn3"
server_key: "373189982aacaafd3eede1cfb3f42ca2"
server_key_iv: "2b3581e59bd04d20cf5f30cbb87af5f7"
server_key_version: "encryptionkey@0"
I20260620 18:32:27.140188 14696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:27.156345 14696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:27.160620 14696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:27.160970 14696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:27.161900 14696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:27.163262 14696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:27.163333 14696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:27.163515 14696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:27.163565 14696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:27.195284 14696 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.3:37935
I20260620 18:32:27.195330 14823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.3:37935 every 8 connection(s)
I20260620 18:32:27.196599 14696 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
I20260620 18:32:27.199695 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 14696
I20260620 18:32:27.199888 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance
I20260620 18:32:27.200428 10764 external_mini_cluster.cc:1428] Setting key 1d1ba3b2008680d714c7cbe599de0688
Jun 20 18:32:27 dist-test-slave-cbn3 krb5kdc[14323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980347, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:32:27.218429 14559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:27.199369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:37839 (local address 127.10.131.62:44529)
0620 18:32:27.199577 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:27.199583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:27.201123 (+ 1540us) server_negotiation.cc:407] Connection header received
0620 18:32:27.203826 (+ 2703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:27.203831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:27.203894 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:27.203972 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:27.205696 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:27.206450 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:27.208689 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:27.209177 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:27.209288 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:27.212695 (+ 3407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:27.212699 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:27.212703 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:27.212735 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:27.215568 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:27.216437 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:27.216441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:27.216444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:27.216561 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:27.216975 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:27.216979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:27.216981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:27.217248 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:27.217361 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:27.217977 (+ 616us) server_negotiation.cc:299] Negotiation successful
0620 18:32:27.218130 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 18:32:27.219435 14826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:27.199924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44529 (local address 127.10.131.3:37839)
0620 18:32:27.200789 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:27.200886 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:27.203560 (+ 2674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:27.204187 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:27.204196 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:27.204965 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:27.205544 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:27.205562 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:27.206622 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:27.206626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:27.208537 (+ 1911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:27.208550 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:27.209427 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:27.209433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:27.209630 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:27.210248 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:27.210268 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:27.212478 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:27.215694 (+ 3216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:27.215702 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:27.215719 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:27.216299 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:27.216659 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:27.216662 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:27.216664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:27.216834 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:27.217352 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:27.217381 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:27.217675 (+ 294us) client_negotiation.cc:769] Sending connection context
0620 18:32:27.217891 (+ 216us) client_negotiation.cc:240] Negotiation successful
0620 18:32:27.218259 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":307,"threads_started":1}
I20260620 18:32:27.220813 14824 heartbeater.cc:347] Connected to a master server at 127.10.131.62:44529
I20260620 18:32:27.221161 14824 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:27.221801 14824 heartbeater.cc:511] Master 127.10.131.62:44529 requested a full tablet report, sending...
I20260620 18:32:27.223109 14368 ts_manager.cc:194] Registered new tserver with Master: 838e5aa38ebe4020b0fe36223a80fec1 (127.10.131.3:37935)
I20260620 18:32:27.223845 14368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.3@KRBTEST.COM'} at 127.10.131.3:37839
I20260620 18:32:27.228425 10764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 18:32:27 dist-test-slave-cbn3 krb5kdc[14323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260620 18:32:27.261523 14557 heartbeater.cc:503] Master 127.10.131.62:44529 was elected leader, sending a full tablet report...
Jun 20 18:32:27 dist-test-slave-cbn3 krb5kdc[14323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.1@KRBTEST.COM
I20260620 18:32:27.277230 14831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:27.261485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47500 (local address 127.10.131.1:45075)
0620 18:32:27.262309 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:27.262319 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:27.262351 (+ 32us) server_negotiation.cc:407] Connection header received
0620 18:32:27.262422 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:27.262428 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:27.262625 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:27.262749 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:27.264421 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:27.265474 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:27.266719 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:27.267326 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:27.267441 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:27.271382 (+ 3941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:27.271393 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:27.271408 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:27.271445 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:27.274356 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:27.275051 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:27.275058 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:27.275060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:27.275206 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:27.275603 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:27.275609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:27.275612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:27.275971 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:27.276285 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:27.276574 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 18:32:27.276869 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":518,"threads_started":1}
Jun 20 18:32:27 dist-test-slave-cbn3 krb5kdc[14323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:32:27.294150 14559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:27.281302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57266 (local address 127.10.131.62:44529)
0620 18:32:27.281536 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:27.281543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:27.281700 (+ 157us) server_negotiation.cc:407] Connection header received
0620 18:32:27.281828 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:27.281832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:27.281901 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:27.281981 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:27.283198 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:27.283943 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:27.285231 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:27.285631 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:27.285728 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:27.289533 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:27.289538 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:27.289541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:27.289575 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:27.291887 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:27.292449 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:27.292452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:27.292454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:27.292577 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:27.292925 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:27.292930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:27.292933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:27.293221 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:27.293402 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:27.293647 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 18:32:27.293796 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260620 18:32:27.294705 14368 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57266
I20260620 18:32:27.295614 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 14429
I20260620 18:32:27.307426 14831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:27.296587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47502 (local address 127.10.131.1:45075)
0620 18:32:27.296806 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:27.296811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:27.296946 (+ 135us) server_negotiation.cc:407] Connection header received
0620 18:32:27.297082 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:27.297086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:27.297146 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:27.297222 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:27.298246 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:27.298893 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:27.300052 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:27.300462 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:27.300555 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:27.302711 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:27.302715 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:27.302718 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:27.302745 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:27.305320 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:27.305860 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:27.305864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:27.305866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:27.305982 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:27.306282 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:27.306285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:27.306287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:27.306499 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:27.306659 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:27.306904 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 18:32:27.307067 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 18:32:27.308084 14552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:27.393091 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 14429
I20260620 18:32:27.409016 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 14563
Jun 20 18:32:27 dist-test-slave-cbn3 krb5kdc[14323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.2@KRBTEST.COM
I20260620 18:32:27.424247 14836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:27.409922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44236 (local address 127.10.131.2:39243)
0620 18:32:27.410407 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:27.410415 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:27.410439 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:32:27.410500 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:27.410506 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:27.410702 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:27.410825 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:27.411946 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:27.413076 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:27.414274 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:27.414781 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:27.414884 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:27.418838 (+ 3954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:27.418848 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:27.418862 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:27.418897 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:27.421863 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:27.422456 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:27.422463 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:27.422465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:27.422581 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:27.422906 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:27.422909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:27.422911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:27.423216 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:27.423420 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:27.423680 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 18:32:27.423860 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":232,"threads_started":1}
I20260620 18:32:27.425603 14686 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:27.507251 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 14563
I20260620 18:32:27.525092 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 14696
Jun 20 18:32:27 dist-test-slave-cbn3 krb5kdc[14323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.3@KRBTEST.COM
I20260620 18:32:27.541115 14838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:27.526342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54634 (local address 127.10.131.3:37935)
0620 18:32:27.527045 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:27.527052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:27.527081 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:32:27.527137 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:27.527143 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:27.527342 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:27.527486 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:27.528650 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:27.529777 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:27.530982 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:27.531533 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:27.531625 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:27.535776 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:27.535787 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:27.535800 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:27.535831 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:27.538571 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:27.539173 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:27.539181 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:27.539183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:27.539306 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:27.539617 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:27.539620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:27.539622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:27.539934 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:27.540280 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:27.540520 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 18:32:27.540704 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":415,"threads_started":1}
I20260620 18:32:27.542316 14819 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:27.627300 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 14696
I20260620 18:32:27.643054 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 14336
I20260620 18:32:27.644162 14398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:27.745254 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 14336
2026-06-20T18:32:27Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3754 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:32:27 dist-test-slave-cbn3 krb5kdc[14843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:32:27 dist-test-slave-cbn3 krb5kdc[14843](info): set up 2 sockets
Jun 20 18:32:27 dist-test-slave-cbn3 krb5kdc[14843](info): commencing operation
krb5kdc: starting...
W20260620 18:32:28.837610 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:32:28 dist-test-slave-cbn3 krb5kdc[14843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:32:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:32:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.131.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
I20260620 18:32:29.028486 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:42099
--webserver_interface=127.10.131.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:34849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:42099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:29.344213 14856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:29.344575 14856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:29.344751 14856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:29.351998 14856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:32:29.352116 14856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:29.352195 14856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:29.352279 14856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:32:29.352365 14856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:32:29.363731 14856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:34849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:42099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:42099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:29.365098 14856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:29.366446 14856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:29.377980 14861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:29.377926 14864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:29.378018 14862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:29.378991 14856 server_base.cc:1034] running on GCE node
I20260620 18:32:29.379696 14856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:29.381042 14856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:29.382237 14856 hybrid_clock.cc:648] HybridClock initialized: now 1781980349382207 us; error 31 us; skew 500 ppm
I20260620 18:32:29.382583 14856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:29 dist-test-slave-cbn3 krb5kdc[14843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980349, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:29.385636 14856 init.cc:377] Logged in from keytab as kudu/127.10.131.62@KRBTEST.COM (short username kudu)
I20260620 18:32:29.386791 14856 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:29.386878 14856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:29.387128 14856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:29.390069 14856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "74a6055bde43485d98dd2e90ec1d0cc2"
format_stamp: "Formatted at 2026-06-20 18:32:29 on dist-test-slave-cbn3"
server_key: "baa16d6f52acb5e3075a6645370b86f5"
server_key_iv: "0f4d7581be13b06790b5dafebd4af027"
server_key_version: "encryptionkey@0"
I20260620 18:32:29.390800 14856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "74a6055bde43485d98dd2e90ec1d0cc2"
format_stamp: "Formatted at 2026-06-20 18:32:29 on dist-test-slave-cbn3"
server_key: "baa16d6f52acb5e3075a6645370b86f5"
server_key_iv: "0f4d7581be13b06790b5dafebd4af027"
server_key_version: "encryptionkey@0"
I20260620 18:32:29.395810 14856 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:32:29.399170 14870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:29.400487 14856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 18:32:29.400632 14856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "74a6055bde43485d98dd2e90ec1d0cc2"
format_stamp: "Formatted at 2026-06-20 18:32:29 on dist-test-slave-cbn3"
server_key: "baa16d6f52acb5e3075a6645370b86f5"
server_key_iv: "0f4d7581be13b06790b5dafebd4af027"
server_key_version: "encryptionkey@0"
I20260620 18:32:29.400817 14856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:29.423285 14856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:29.427383 14856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:29.427630 14856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:29.445936 14856 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:42099
I20260620 18:32:29.445961 14921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:42099 every 8 connection(s)
I20260620 18:32:29.446853 14856 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:32:29.450610 14922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:29.456734 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 14856
I20260620 18:32:29.457005 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:32:29.457635 10764 external_mini_cluster.cc:1428] Setting key 908b474578869fc92d704c6f1d21acdf
I20260620 18:32:29.461226 14922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74a6055bde43485d98dd2e90ec1d0cc2: Bootstrap starting.
I20260620 18:32:29.464881 14922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74a6055bde43485d98dd2e90ec1d0cc2: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:29.465943 14922 log.cc:824] T 00000000000000000000000000000000 P 74a6055bde43485d98dd2e90ec1d0cc2: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:29.468768 14922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74a6055bde43485d98dd2e90ec1d0cc2: No bootstrap required, opened a new log
Jun 20 18:32:29 dist-test-slave-cbn3 krb5kdc[14843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:32:29.479526 14922 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74a6055bde43485d98dd2e90ec1d0cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a6055bde43485d98dd2e90ec1d0cc2" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 42099 } }
I20260620 18:32:29.480016 14922 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74a6055bde43485d98dd2e90ec1d0cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:29.480130 14922 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74a6055bde43485d98dd2e90ec1d0cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a6055bde43485d98dd2e90ec1d0cc2, State: Initialized, Role: FOLLOWER
I20260620 18:32:29.480764 14922 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74a6055bde43485d98dd2e90ec1d0cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a6055bde43485d98dd2e90ec1d0cc2" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 42099 } }
I20260620 18:32:29.481089 14922 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74a6055bde43485d98dd2e90ec1d0cc2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:32:29.481243 14922 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74a6055bde43485d98dd2e90ec1d0cc2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:32:29.481423 14922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74a6055bde43485d98dd2e90ec1d0cc2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:29.481974 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:29.459577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43436 (local address 127.10.131.62:42099)
0620 18:32:29.460333 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:29.460344 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:29.460383 (+ 39us) server_negotiation.cc:407] Connection header received
0620 18:32:29.462131 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:29.462154 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:29.462534 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:29.462861 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:29.464758 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:29.466212 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:29.467917 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:29.468569 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:29.468703 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:29.473316 (+ 4613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:29.473325 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:29.473346 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:29.473403 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:29.477461 (+ 4058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:29.478138 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:29.478147 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:29.478155 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:29.478352 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:29.478694 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:29.478699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:29.478703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:29.479400 (+ 697us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:29.479666 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:29.480074 (+ 408us) server_negotiation.cc:299] Negotiation successful
0620 18:32:29.480511 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":350,"threads_started":1}
I20260620 18:32:29.483980 14922 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74a6055bde43485d98dd2e90ec1d0cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a6055bde43485d98dd2e90ec1d0cc2" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 42099 } }
I20260620 18:32:29.484474 14922 leader_election.cc:302] T 00000000000000000000000000000000 P 74a6055bde43485d98dd2e90ec1d0cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74a6055bde43485d98dd2e90ec1d0cc2; no voters:
I20260620 18:32:29.485085 14922 leader_election.cc:288] T 00000000000000000000000000000000 P 74a6055bde43485d98dd2e90ec1d0cc2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:32:29.485180 14927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74a6055bde43485d98dd2e90ec1d0cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:29.485843 14927 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74a6055bde43485d98dd2e90ec1d0cc2 [term 1 LEADER]: Becoming Leader. State: Replica: 74a6055bde43485d98dd2e90ec1d0cc2, State: Running, Role: LEADER
I20260620 18:32:29.486470 14927 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74a6055bde43485d98dd2e90ec1d0cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a6055bde43485d98dd2e90ec1d0cc2" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 42099 } }
I20260620 18:32:29.487404 14922 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74a6055bde43485d98dd2e90ec1d0cc2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:32:29.489581 14928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74a6055bde43485d98dd2e90ec1d0cc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74a6055bde43485d98dd2e90ec1d0cc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a6055bde43485d98dd2e90ec1d0cc2" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 42099 } } }
I20260620 18:32:29.489650 14929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74a6055bde43485d98dd2e90ec1d0cc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74a6055bde43485d98dd2e90ec1d0cc2. Latest consensus state: current_term: 1 leader_uuid: "74a6055bde43485d98dd2e90ec1d0cc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a6055bde43485d98dd2e90ec1d0cc2" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 42099 } } }
I20260620 18:32:29.490001 14928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74a6055bde43485d98dd2e90ec1d0cc2 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:29.490059 14929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74a6055bde43485d98dd2e90ec1d0cc2 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:29.496974 14933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:32:29.502207 14933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:32:29.511534 14933 catalog_manager.cc:1347] Generated new cluster ID: 2da923b4c836447f84fb360de3bc5d3f
I20260620 18:32:29.511657 14933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:32:29.543408 14933 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:32:29.545320 14933 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:32:29.560359 14933 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74a6055bde43485d98dd2e90ec1d0cc2: Generated new TSK 0
I20260620 18:32:29.561420 14933 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.131.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab.
Entry for principal kudu/127.10.131.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab.
I20260620 18:32:29.621055 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.1:0
--local_ip_for_outbound_sockets=127.10.131.1
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:42099
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:34849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:29.944686 14948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:29.944977 14948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:29.945139 14948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:29.952513 14948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:29.952613 14948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:29.952800 14948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.1
I20260620 18:32:29.962587 14948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:34849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:42099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.1
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:29.963797 14948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:29.965083 14948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:29.977525 14953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:29.977489 14954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:29.977449 14956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:29.978626 14948 server_base.cc:1034] running on GCE node
I20260620 18:32:29.979215 14948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:29.980096 14948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:29.981285 14948 hybrid_clock.cc:648] HybridClock initialized: now 1781980349981248 us; error 33 us; skew 500 ppm
I20260620 18:32:29.981594 14948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:29 dist-test-slave-cbn3 krb5kdc[14843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980349, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:29.984629 14948 init.cc:377] Logged in from keytab as kudu/127.10.131.1@KRBTEST.COM (short username kudu)
I20260620 18:32:29.985692 14948 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:29.985774 14948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:29.986032 14948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:29.988979 14948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-0/data/instance:
uuid: "25051c5c5ac74f0e94343eb702029a06"
format_stamp: "Formatted at 2026-06-20 18:32:29 on dist-test-slave-cbn3"
server_key: "90e7ba32db22cbc3a5f3ee641d8671dd"
server_key_iv: "3e4f0e4d855766cf151914947cdb9a59"
server_key_version: "encryptionkey@0"
I20260620 18:32:29.989699 14948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance:
uuid: "25051c5c5ac74f0e94343eb702029a06"
format_stamp: "Formatted at 2026-06-20 18:32:29 on dist-test-slave-cbn3"
server_key: "90e7ba32db22cbc3a5f3ee641d8671dd"
server_key_iv: "3e4f0e4d855766cf151914947cdb9a59"
server_key_version: "encryptionkey@0"
I20260620 18:32:29.994488 14948 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 18:32:29.997959 14962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:29.999275 14948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:32:29.999402 14948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-0/wal
uuid: "25051c5c5ac74f0e94343eb702029a06"
format_stamp: "Formatted at 2026-06-20 18:32:29 on dist-test-slave-cbn3"
server_key: "90e7ba32db22cbc3a5f3ee641d8671dd"
server_key_iv: "3e4f0e4d855766cf151914947cdb9a59"
server_key_version: "encryptionkey@0"
I20260620 18:32:29.999572 14948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:30.032261 14948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:30.037503 14948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:30.037745 14948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:30.038690 14948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:30.040298 14948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:30.040398 14948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:30.040473 14948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:30.040530 14948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:30.074755 14948 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.1:35929
I20260620 18:32:30.074856 15074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.1:35929 every 8 connection(s)
I20260620 18:32:30.075639 14948 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
I20260620 18:32:30.082098 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 14948
I20260620 18:32:30.082324 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance
I20260620 18:32:30.082965 10764 external_mini_cluster.cc:1428] Setting key bacd9018f108e1e98fd9c44e37ac5bf7
Jun 20 18:32:30 dist-test-slave-cbn3 krb5kdc[14843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980349, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:32:30.096122 15077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:30.078232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:48587 (local address 127.10.131.62:42099)
0620 18:32:30.078692 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:30.078698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:30.079647 (+ 949us) server_negotiation.cc:407] Connection header received
0620 18:32:30.081445 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:30.081450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:30.081520 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:30.081621 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:30.083206 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:30.084205 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:30.085878 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:30.086278 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:30.086415 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:30.090378 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:30.090383 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:30.090386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:30.090417 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:30.093029 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:30.093778 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:30.093783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:30.093785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:30.093903 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:30.094369 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:30.094374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:30.094376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:30.094620 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:30.094780 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:30.095480 (+ 700us) server_negotiation.cc:299] Negotiation successful
0620 18:32:30.095702 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":211,"threads_started":1}
I20260620 18:32:30.097316 15078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:30.078822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:42099 (local address 127.10.131.1:48587)
0620 18:32:30.079491 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:30.079532 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:30.081199 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:30.081810 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:30.081817 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:30.082373 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:30.082976 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:30.082993 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:30.084374 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:30.084384 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:30.085718 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:30.085730 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:30.086515 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:30.086522 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:30.086742 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:30.087479 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:30.087510 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:30.090144 (+ 2634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:30.093176 (+ 3032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:30.093185 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:30.093202 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:30.093636 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:30.094027 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:30.094032 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:30.094035 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:30.094236 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:30.094747 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:30.094755 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:30.095093 (+ 338us) client_negotiation.cc:769] Sending connection context
0620 18:32:30.095368 (+ 275us) client_negotiation.cc:240] Negotiation successful
0620 18:32:30.095894 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":265,"threads_started":1}
I20260620 18:32:30.099016 15075 heartbeater.cc:347] Connected to a master server at 127.10.131.62:42099
I20260620 18:32:30.099365 15075 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:30.100214 15075 heartbeater.cc:511] Master 127.10.131.62:42099 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.131.2@KRBTEST.COM; defaulting to no policy
I20260620 18:32:30.102586 14887 ts_manager.cc:194] Registered new tserver with Master: 25051c5c5ac74f0e94343eb702029a06 (127.10.131.1:35929)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.2@KRBTEST.COM" created.
I20260620 18:32:30.104640 14887 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.1@KRBTEST.COM'} at 127.10.131.1:48587
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab.
Entry for principal kudu/127.10.131.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab.
I20260620 18:32:30.127244 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.2:0
--local_ip_for_outbound_sockets=127.10.131.2
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:42099
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:34849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:30.442683 15081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:30.442971 15081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:30.443138 15081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:30.450917 15081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:30.451013 15081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:30.451211 15081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.2
I20260620 18:32:30.460481 15081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:34849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:42099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.2
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:30.461663 15081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:30.462882 15081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:30.475432 15087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:30.475479 15086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:30.475337 15089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:30.476594 15081 server_base.cc:1034] running on GCE node
I20260620 18:32:30.477308 15081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:30.478250 15081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:30.479496 15081 hybrid_clock.cc:648] HybridClock initialized: now 1781980350479430 us; error 58 us; skew 500 ppm
I20260620 18:32:30.479861 15081 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:30 dist-test-slave-cbn3 krb5kdc[14843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980350, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:30.483130 15081 init.cc:377] Logged in from keytab as kudu/127.10.131.2@KRBTEST.COM (short username kudu)
I20260620 18:32:30.484556 15081 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:30.484650 15081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:30.485023 15081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:30.487957 15081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-1/data/instance:
uuid: "eca4f5b603bb4f87811018aa4ec07abb"
format_stamp: "Formatted at 2026-06-20 18:32:30 on dist-test-slave-cbn3"
server_key: "a3f18d324e12f38f664aae7f15babc6f"
server_key_iv: "aa578b67f2db29d5cb4626019b4f6d6f"
server_key_version: "encryptionkey@0"
I20260620 18:32:30.488723 15081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance:
uuid: "eca4f5b603bb4f87811018aa4ec07abb"
format_stamp: "Formatted at 2026-06-20 18:32:30 on dist-test-slave-cbn3"
server_key: "a3f18d324e12f38f664aae7f15babc6f"
server_key_iv: "aa578b67f2db29d5cb4626019b4f6d6f"
server_key_version: "encryptionkey@0"
I20260620 18:32:30.493716 15081 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:32:30.497463 15095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:30.498781 15081 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 18:32:30.498941 15081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-1/wal
uuid: "eca4f5b603bb4f87811018aa4ec07abb"
format_stamp: "Formatted at 2026-06-20 18:32:30 on dist-test-slave-cbn3"
server_key: "a3f18d324e12f38f664aae7f15babc6f"
server_key_iv: "aa578b67f2db29d5cb4626019b4f6d6f"
server_key_version: "encryptionkey@0"
I20260620 18:32:30.499169 15081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:30.511868 15081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:30.515892 15081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:30.516170 15081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:30.517217 15081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:30.518517 15081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:30.518601 15081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:30.518720 15081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:30.518828 15081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:30.553362 15081 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.2:39165
I20260620 18:32:30.553418 15207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.2:39165 every 8 connection(s)
I20260620 18:32:30.554224 15081 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
I20260620 18:32:30.555379 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 15081
I20260620 18:32:30.555593 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance
I20260620 18:32:30.556125 10764 external_mini_cluster.cc:1428] Setting key 89dba7186438d9a54c6084553f909645
Jun 20 18:32:30 dist-test-slave-cbn3 krb5kdc[14843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980350, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.131.3@KRBTEST.COM; defaulting to no policy
I20260620 18:32:30.576109 15077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:30.556892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:43291 (local address 127.10.131.62:42099)
0620 18:32:30.557082 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:30.557087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:30.558412 (+ 1325us) server_negotiation.cc:407] Connection header received
0620 18:32:30.560124 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:30.560130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:30.560220 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:30.560332 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:30.561853 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:30.562518 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:30.564708 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:30.565189 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:30.565323 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:30.568952 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:30.568962 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:30.568970 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:30.569014 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:30.572822 (+ 3808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:30.573646 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:30.573654 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:30.573657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:30.573820 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:30.574288 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:30.574294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:30.574297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:30.574590 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:30.574751 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:30.575539 (+ 788us) server_negotiation.cc:299] Negotiation successful
0620 18:32:30.575771 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:32:30.577418 15210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:30.557465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:42099 (local address 127.10.131.2:43291)
0620 18:32:30.558267 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:30.558305 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:30.559873 (+ 1568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:30.560558 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:30.560566 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:30.561131 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:30.561668 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:30.561685 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:30.562675 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:30.562685 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:30.564560 (+ 1875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:30.564572 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:30.565418 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:30.565425 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:30.565638 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:30.566327 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:30.566354 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:30.568682 (+ 2328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:30.573040 (+ 4358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:30.573047 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:30.573065 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:30.573510 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:30.573964 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:30.573967 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:30.573970 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:30.574169 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:30.574723 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:30.574731 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:30.575086 (+ 355us) client_negotiation.cc:769] Sending connection context
0620 18:32:30.575435 (+ 349us) client_negotiation.cc:240] Negotiation successful
0620 18:32:30.576025 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":333,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.3@KRBTEST.COM" created.
I20260620 18:32:30.579144 15208 heartbeater.cc:347] Connected to a master server at 127.10.131.62:42099
I20260620 18:32:30.579507 15208 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:30.580224 15208 heartbeater.cc:511] Master 127.10.131.62:42099 requested a full tablet report, sending...
I20260620 18:32:30.581823 14887 ts_manager.cc:194] Registered new tserver with Master: eca4f5b603bb4f87811018aa4ec07abb (127.10.131.2:39165)
I20260620 18:32:30.582706 14887 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.2@KRBTEST.COM'} at 127.10.131.2:43291
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab.
Entry for principal kudu/127.10.131.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab.
I20260620 18:32:30.602509 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.3:0
--local_ip_for_outbound_sockets=127.10.131.3
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:42099
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:34849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:30.921463 15213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:30.921736 15213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:30.921913 15213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:30.929143 15213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:30.929243 15213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:30.929440 15213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.3
I20260620 18:32:30.939173 15213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:34849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:42099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.3
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:30.940593 15213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:30.941890 15213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:30.954499 15219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:30.954459 15218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:30.954744 15221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:30.955299 15213 server_base.cc:1034] running on GCE node
I20260620 18:32:30.955917 15213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:30.956881 15213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:30.958125 15213 hybrid_clock.cc:648] HybridClock initialized: now 1781980350958082 us; error 50 us; skew 500 ppm
I20260620 18:32:30.958453 15213 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:30 dist-test-slave-cbn3 krb5kdc[14843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980350, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:30.964751 15213 init.cc:377] Logged in from keytab as kudu/127.10.131.3@KRBTEST.COM (short username kudu)
I20260620 18:32:30.965993 15213 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:30.966080 15213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:30.966326 15213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:30.969220 15213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-2/data/instance:
uuid: "feb95057296a4c968db65b492934213b"
format_stamp: "Formatted at 2026-06-20 18:32:30 on dist-test-slave-cbn3"
server_key: "fd5668cbb9d0bae838d4baa1fb0deeb2"
server_key_iv: "e53414d3456c475d111dd3e745fc9f60"
server_key_version: "encryptionkey@0"
I20260620 18:32:30.969954 15213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance:
uuid: "feb95057296a4c968db65b492934213b"
format_stamp: "Formatted at 2026-06-20 18:32:30 on dist-test-slave-cbn3"
server_key: "fd5668cbb9d0bae838d4baa1fb0deeb2"
server_key_iv: "e53414d3456c475d111dd3e745fc9f60"
server_key_version: "encryptionkey@0"
I20260620 18:32:30.974757 15213 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:32:30.978420 15227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:30.979811 15213 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:32:30.979950 15213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-2/wal
uuid: "feb95057296a4c968db65b492934213b"
format_stamp: "Formatted at 2026-06-20 18:32:30 on dist-test-slave-cbn3"
server_key: "fd5668cbb9d0bae838d4baa1fb0deeb2"
server_key_iv: "e53414d3456c475d111dd3e745fc9f60"
server_key_version: "encryptionkey@0"
I20260620 18:32:30.980136 15213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:31.007496 15213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:31.011554 15213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:31.011782 15213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:31.012701 15213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:31.014164 15213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:31.014252 15213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:31.014323 15213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:31.014379 15213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:31.056241 15213 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.3:40527
I20260620 18:32:31.056360 15339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.3:40527 every 8 connection(s)
I20260620 18:32:31.057307 15213 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
I20260620 18:32:31.063727 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 15213
I20260620 18:32:31.063958 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance
I20260620 18:32:31.064385 10764 external_mini_cluster.cc:1428] Setting key d77c42e193fa90c212fe908bd127c498
Jun 20 18:32:31 dist-test-slave-cbn3 krb5kdc[14843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980350, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:32:31.082516 15077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:31.060225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:55463 (local address 127.10.131.62:42099)
0620 18:32:31.060430 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:31.060438 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:31.061953 (+ 1515us) server_negotiation.cc:407] Connection header received
0620 18:32:31.065124 (+ 3171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:31.065133 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:31.065238 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:31.065328 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:31.067885 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.068655 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.071569 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.071961 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.072068 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:31.075830 (+ 3762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:31.075837 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:31.075842 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:31.075880 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:31.079924 (+ 4044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:31.080609 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:31.080613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:31.080615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:31.080729 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:31.081153 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:31.081156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:31.081158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:31.081404 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:31.081531 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:31.082113 (+ 582us) server_negotiation.cc:299] Negotiation successful
0620 18:32:31.082264 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 18:32:31.083428 15342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:31.060799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:42099 (local address 127.10.131.3:55463)
0620 18:32:31.061747 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:31.061806 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:31.064675 (+ 2869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:31.065894 (+ 1219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:31.065908 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:31.066753 (+ 845us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:31.067634 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:31.067661 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.068809 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.068817 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:31.071381 (+ 2564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:31.071400 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.072164 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.072170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:31.072357 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:31.073161 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:31.073184 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:31.075626 (+ 2442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:31.080072 (+ 4446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:31.080078 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:31.080089 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:31.080493 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:31.080817 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:31.080820 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:31.080822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:31.081054 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:31.081507 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:31.081512 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:31.081789 (+ 277us) client_negotiation.cc:769] Sending connection context
0620 18:32:31.082021 (+ 232us) client_negotiation.cc:240] Negotiation successful
0620 18:32:31.082374 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":362,"threads_started":1}
I20260620 18:32:31.084798 15340 heartbeater.cc:347] Connected to a master server at 127.10.131.62:42099
I20260620 18:32:31.085129 15340 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:31.085769 15340 heartbeater.cc:511] Master 127.10.131.62:42099 requested a full tablet report, sending...
I20260620 18:32:31.087306 14887 ts_manager.cc:194] Registered new tserver with Master: feb95057296a4c968db65b492934213b (127.10.131.3:40527)
I20260620 18:32:31.088156 14887 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.3@KRBTEST.COM'} at 127.10.131.3:55463
I20260620 18:32:31.091746 10764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:32:31.108384 15075 heartbeater.cc:503] Master 127.10.131.62:42099 was elected leader, sending a full tablet report...
I20260620 18:32:31.108580 15077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:31.096163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43448 (local address 127.10.131.62:42099)
0620 18:32:31.096357 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:31.096365 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:31.096998 (+ 633us) server_negotiation.cc:407] Connection header received
0620 18:32:31.097142 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:31.097146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:31.097207 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:31.097289 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:31.098762 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.099466 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.100658 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.101158 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.101252 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:31.103677 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:31.103681 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:31.103684 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:31.103713 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:31.106353 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:31.106935 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:31.106940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:31.106942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:31.107071 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:31.107362 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:31.107365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:31.107367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:31.107653 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:31.107822 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:31.108051 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 18:32:31.108254 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 18:32:31.112314 14886 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43448:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:32:31.119108 14886 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:32:31.162863 15356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:31.130817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54406 (local address 127.10.131.1:35929)
0620 18:32:31.154417 (+ 23600us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:31.154429 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:31.154464 (+ 35us) server_negotiation.cc:407] Connection header received
0620 18:32:31.154588 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:31.154599 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:31.154886 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:31.155105 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:31.155815 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.157431 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.160349 (+ 2918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.161704 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.161867 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:31.162098 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:31.162344 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 18:32:31.162514 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23375,"thread_start_us":23187,"threads_started":1}
I20260620 18:32:31.162902 15352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:31.130258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:35929 (local address 127.0.0.1:54406)
0620 18:32:31.140763 (+ 10505us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:31.140824 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:31.141077 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:31.155124 (+ 14047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:31.155130 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:31.155177 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:31.155649 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:31.155668 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.157550 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.157556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:31.160166 (+ 2610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:31.160180 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.161919 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.161925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:31.162091 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:31.162181 (+ 90us) client_negotiation.cc:769] Sending connection context
0620 18:32:31.162297 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 18:32:31.162514 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10174,"mutex_wait_us":54,"thread_start_us":10009,"threads_started":1}
I20260620 18:32:31.163013 15355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:31.130385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50820 (local address 127.10.131.3:40527)
0620 18:32:31.154984 (+ 24599us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:31.154989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:31.155013 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:32:31.155122 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:31.155129 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:31.155374 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:31.155572 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:31.156521 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.158065 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.160289 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.161320 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.161491 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:31.161731 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:31.162435 (+ 704us) server_negotiation.cc:299] Negotiation successful
0620 18:32:31.162628 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24445,"thread_start_us":24294,"threads_started":1}
I20260620 18:32:31.163836 15353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:31.130298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:40527 (local address 127.0.0.1:50820)
0620 18:32:31.153371 (+ 23073us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:31.153395 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:31.153535 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:31.155846 (+ 2311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:31.155851 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:31.155879 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:31.156292 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:31.156305 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.158195 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.158199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:31.160131 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:31.160142 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.161863 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.161869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:31.162066 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:31.162150 (+ 84us) client_negotiation.cc:769] Sending connection context
0620 18:32:31.162281 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 18:32:31.163417 (+ 1136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22894,"mutex_wait_us":1024,"thread_start_us":22228,"threads_started":1}
I20260620 18:32:31.164808 15354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:31.130612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:39165 (local address 127.0.0.1:58854)
0620 18:32:31.154649 (+ 24037us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:31.154666 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:31.154788 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:31.155507 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:31.155512 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:31.155541 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:31.155993 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:31.156011 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.157484 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.157490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:31.161256 (+ 3766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:31.161271 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.163352 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.163358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:31.163549 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:31.163582 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:32:31.163670 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 18:32:31.163818 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23898,"mutex_wait_us":1446,"thread_start_us":23764,"threads_started":1}
I20260620 18:32:31.165419 15357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:31.131123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58854 (local address 127.10.131.2:39165)
0620 18:32:31.154136 (+ 23013us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:31.154145 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:31.154923 (+ 778us) server_negotiation.cc:407] Connection header received
0620 18:32:31.155030 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:31.155039 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:31.155295 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:31.155519 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:31.156207 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.157351 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.161430 (+ 4079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.162840 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.162996 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:31.163190 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:31.163928 (+ 738us) server_negotiation.cc:299] Negotiation successful
0620 18:32:31.165189 (+ 1261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22845,"thread_start_us":23023,"threads_started":1}
I20260620 18:32:31.167367 15010 tablet_service.cc:1467] Processing CreateTablet for tablet 56fd6eb46cf44f8eb11ed21a165c02d8 (DEFAULT_TABLE table=test-table [id=c14711aba4974d76a3047e9d2382f9ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:31.167691 15275 tablet_service.cc:1467] Processing CreateTablet for tablet 56fd6eb46cf44f8eb11ed21a165c02d8 (DEFAULT_TABLE table=test-table [id=c14711aba4974d76a3047e9d2382f9ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:31.168807 15275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56fd6eb46cf44f8eb11ed21a165c02d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:31.168916 15010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56fd6eb46cf44f8eb11ed21a165c02d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:31.168699 15143 tablet_service.cc:1467] Processing CreateTablet for tablet 56fd6eb46cf44f8eb11ed21a165c02d8 (DEFAULT_TABLE table=test-table [id=c14711aba4974d76a3047e9d2382f9ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:31.169911 15143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56fd6eb46cf44f8eb11ed21a165c02d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:31.179638 15358 tablet_bootstrap.cc:492] T 56fd6eb46cf44f8eb11ed21a165c02d8 P feb95057296a4c968db65b492934213b: Bootstrap starting.
I20260620 18:32:31.182787 15358 tablet_bootstrap.cc:654] T 56fd6eb46cf44f8eb11ed21a165c02d8 P feb95057296a4c968db65b492934213b: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:31.184026 15359 tablet_bootstrap.cc:492] T 56fd6eb46cf44f8eb11ed21a165c02d8 P eca4f5b603bb4f87811018aa4ec07abb: Bootstrap starting.
I20260620 18:32:31.184151 15358 log.cc:824] T 56fd6eb46cf44f8eb11ed21a165c02d8 P feb95057296a4c968db65b492934213b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:31.187479 15358 tablet_bootstrap.cc:492] T 56fd6eb46cf44f8eb11ed21a165c02d8 P feb95057296a4c968db65b492934213b: No bootstrap required, opened a new log
I20260620 18:32:31.187675 15359 tablet_bootstrap.cc:654] T 56fd6eb46cf44f8eb11ed21a165c02d8 P eca4f5b603bb4f87811018aa4ec07abb: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:31.187740 15358 ts_tablet_manager.cc:1397] T 56fd6eb46cf44f8eb11ed21a165c02d8 P feb95057296a4c968db65b492934213b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260620 18:32:31.188891 15359 log.cc:824] T 56fd6eb46cf44f8eb11ed21a165c02d8 P eca4f5b603bb4f87811018aa4ec07abb: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:31.188980 15361 tablet_bootstrap.cc:492] T 56fd6eb46cf44f8eb11ed21a165c02d8 P 25051c5c5ac74f0e94343eb702029a06: Bootstrap starting.
I20260620 18:32:31.192034 15359 tablet_bootstrap.cc:492] T 56fd6eb46cf44f8eb11ed21a165c02d8 P eca4f5b603bb4f87811018aa4ec07abb: No bootstrap required, opened a new log
I20260620 18:32:31.192273 15359 ts_tablet_manager.cc:1397] T 56fd6eb46cf44f8eb11ed21a165c02d8 P eca4f5b603bb4f87811018aa4ec07abb: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260620 18:32:31.192510 15361 tablet_bootstrap.cc:654] T 56fd6eb46cf44f8eb11ed21a165c02d8 P 25051c5c5ac74f0e94343eb702029a06: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:31.193673 15361 log.cc:824] T 56fd6eb46cf44f8eb11ed21a165c02d8 P 25051c5c5ac74f0e94343eb702029a06: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:31.196448 15361 tablet_bootstrap.cc:492] T 56fd6eb46cf44f8eb11ed21a165c02d8 P 25051c5c5ac74f0e94343eb702029a06: No bootstrap required, opened a new log
I20260620 18:32:31.196731 15361 ts_tablet_manager.cc:1397] T 56fd6eb46cf44f8eb11ed21a165c02d8 P 25051c5c5ac74f0e94343eb702029a06: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.005s
I20260620 18:32:31.198163 15358 raft_consensus.cc:348] T 56fd6eb46cf44f8eb11ed21a165c02d8 P feb95057296a4c968db65b492934213b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb95057296a4c968db65b492934213b" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40527 } } peers { permanent_uuid: "25051c5c5ac74f0e94343eb702029a06" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35929 } } peers { permanent_uuid: "eca4f5b603bb4f87811018aa4ec07abb" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39165 } }
I20260620 18:32:31.198583 15358 raft_consensus.cc:374] T 56fd6eb46cf44f8eb11ed21a165c02d8 P feb95057296a4c968db65b492934213b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:31.198678 15358 raft_consensus.cc:729] T 56fd6eb46cf44f8eb11ed21a165c02d8 P feb95057296a4c968db65b492934213b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb95057296a4c968db65b492934213b, State: Initialized, Role: FOLLOWER
I20260620 18:32:31.199267 15358 consensus_queue.cc:260] T 56fd6eb46cf44f8eb11ed21a165c02d8 P feb95057296a4c968db65b492934213b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb95057296a4c968db65b492934213b" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40527 } } peers { permanent_uuid: "25051c5c5ac74f0e94343eb702029a06" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35929 } } peers { permanent_uuid: "eca4f5b603bb4f87811018aa4ec07abb" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39165 } }
I20260620 18:32:31.200392 15340 heartbeater.cc:503] Master 127.10.131.62:42099 was elected leader, sending a full tablet report...
I20260620 18:32:31.200991 15358 ts_tablet_manager.cc:1428] T 56fd6eb46cf44f8eb11ed21a165c02d8 P feb95057296a4c968db65b492934213b: Time spent starting tablet: real 0.013s user 0.013s sys 0.002s
I20260620 18:32:31.202507 15359 raft_consensus.cc:348] T 56fd6eb46cf44f8eb11ed21a165c02d8 P eca4f5b603bb4f87811018aa4ec07abb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb95057296a4c968db65b492934213b" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40527 } } peers { permanent_uuid: "25051c5c5ac74f0e94343eb702029a06" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35929 } } peers { permanent_uuid: "eca4f5b603bb4f87811018aa4ec07abb" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39165 } }
I20260620 18:32:31.202924 15359 raft_consensus.cc:374] T 56fd6eb46cf44f8eb11ed21a165c02d8 P eca4f5b603bb4f87811018aa4ec07abb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:31.203266 15359 raft_consensus.cc:729] T 56fd6eb46cf44f8eb11ed21a165c02d8 P eca4f5b603bb4f87811018aa4ec07abb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eca4f5b603bb4f87811018aa4ec07abb, State: Initialized, Role: FOLLOWER
I20260620 18:32:31.204046 15359 consensus_queue.cc:260] T 56fd6eb46cf44f8eb11ed21a165c02d8 P eca4f5b603bb4f87811018aa4ec07abb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb95057296a4c968db65b492934213b" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40527 } } peers { permanent_uuid: "25051c5c5ac74f0e94343eb702029a06" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35929 } } peers { permanent_uuid: "eca4f5b603bb4f87811018aa4ec07abb" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39165 } }
I20260620 18:32:31.205368 15208 heartbeater.cc:503] Master 127.10.131.62:42099 was elected leader, sending a full tablet report...
I20260620 18:32:31.205804 15359 ts_tablet_manager.cc:1428] T 56fd6eb46cf44f8eb11ed21a165c02d8 P eca4f5b603bb4f87811018aa4ec07abb: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260620 18:32:31.207548 15361 raft_consensus.cc:348] T 56fd6eb46cf44f8eb11ed21a165c02d8 P 25051c5c5ac74f0e94343eb702029a06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb95057296a4c968db65b492934213b" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40527 } } peers { permanent_uuid: "25051c5c5ac74f0e94343eb702029a06" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35929 } } peers { permanent_uuid: "eca4f5b603bb4f87811018aa4ec07abb" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39165 } }
I20260620 18:32:31.207988 15361 raft_consensus.cc:374] T 56fd6eb46cf44f8eb11ed21a165c02d8 P 25051c5c5ac74f0e94343eb702029a06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:31.208099 15361 raft_consensus.cc:729] T 56fd6eb46cf44f8eb11ed21a165c02d8 P 25051c5c5ac74f0e94343eb702029a06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25051c5c5ac74f0e94343eb702029a06, State: Initialized, Role: FOLLOWER
I20260620 18:32:31.208719 15361 consensus_queue.cc:260] T 56fd6eb46cf44f8eb11ed21a165c02d8 P 25051c5c5ac74f0e94343eb702029a06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb95057296a4c968db65b492934213b" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40527 } } peers { permanent_uuid: "25051c5c5ac74f0e94343eb702029a06" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35929 } } peers { permanent_uuid: "eca4f5b603bb4f87811018aa4ec07abb" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39165 } }
I20260620 18:32:31.210395 15361 ts_tablet_manager.cc:1428] T 56fd6eb46cf44f8eb11ed21a165c02d8 P 25051c5c5ac74f0e94343eb702029a06: Time spent starting tablet: real 0.014s user 0.011s sys 0.002s
W20260620 18:32:31.306354 15209 tablet.cc:2369] T 56fd6eb46cf44f8eb11ed21a165c02d8 P eca4f5b603bb4f87811018aa4ec07abb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:32:31.308868 15341 tablet.cc:2369] T 56fd6eb46cf44f8eb11ed21a165c02d8 P feb95057296a4c968db65b492934213b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:32:31.327976 15076 tablet.cc:2369] T 56fd6eb46cf44f8eb11ed21a165c02d8 P 25051c5c5ac74f0e94343eb702029a06: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:32:31.462841 15365 raft_consensus.cc:482] T 56fd6eb46cf44f8eb11ed21a165c02d8 P eca4f5b603bb4f87811018aa4ec07abb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:32:31.463186 15365 raft_consensus.cc:504] T 56fd6eb46cf44f8eb11ed21a165c02d8 P eca4f5b603bb4f87811018aa4ec07abb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb95057296a4c968db65b492934213b" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40527 } } peers { permanent_uuid: "25051c5c5ac74f0e94343eb702029a06" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35929 } } peers { permanent_uuid: "eca4f5b603bb4f87811018aa4ec07abb" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39165 } }
I20260620 18:32:31.464905 15365 leader_election.cc:288] T 56fd6eb46cf44f8eb11ed21a165c02d8 P eca4f5b603bb4f87811018aa4ec07abb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers feb95057296a4c968db65b492934213b (127.10.131.3:40527), 25051c5c5ac74f0e94343eb702029a06 (127.10.131.1:35929)
I20260620 18:32:31.471639 15356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:31.465623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:36151 (local address 127.10.131.1:35929)
0620 18:32:31.465857 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:31.465866 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:31.466517 (+ 651us) server_negotiation.cc:407] Connection header received
0620 18:32:31.466712 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:31.466716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:31.466771 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:31.466866 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:31.467470 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.468169 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.469938 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.470679 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.470854 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:31.470930 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:31.471336 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 18:32:31.471468 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 18:32:31.471644 15355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:31.465377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:53241 (local address 127.10.131.3:40527)
0620 18:32:31.465624 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:31.465634 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:31.466301 (+ 667us) server_negotiation.cc:407] Connection header received
0620 18:32:31.466537 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:31.466542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:31.466599 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:31.466693 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:31.467243 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.467962 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.469803 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.470573 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.470750 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:31.470824 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:31.471273 (+ 449us) server_negotiation.cc:299] Negotiation successful
0620 18:32:31.471396 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 18:32:31.471758 15367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:31.465200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:40527 (local address 127.10.131.2:53241)
0620 18:32:31.466171 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:31.466190 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:31.466374 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:31.466714 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:31.466718 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:31.466743 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:31.467095 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:31.467108 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.468093 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.468100 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:31.469636 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:31.469647 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.470773 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.470779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:31.470954 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:31.470976 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:32:31.471231 (+ 255us) client_negotiation.cc:240] Negotiation successful
0620 18:32:31.471366 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"mutex_wait_us":1,"thread_start_us":635,"threads_started":1}
I20260620 18:32:31.472270 15368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:31.465232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:35929 (local address 127.10.131.2:36151)
0620 18:32:31.466403 (+ 1171us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:31.466416 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:31.466563 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:31.466885 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:31.466889 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:31.466916 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:31.467300 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:31.467309 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.468307 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.468313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:31.469742 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:31.469754 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.470893 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.470899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:31.471130 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:31.471171 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 18:32:31.471896 (+ 725us) client_negotiation.cc:240] Negotiation successful
0620 18:32:31.472021 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1069,"thread_start_us":864,"threads_started":1}
I20260620 18:32:31.472746 15295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56fd6eb46cf44f8eb11ed21a165c02d8" candidate_uuid: "eca4f5b603bb4f87811018aa4ec07abb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb95057296a4c968db65b492934213b" is_pre_election: true
I20260620 18:32:31.473265 15295 raft_consensus.cc:2454] T 56fd6eb46cf44f8eb11ed21a165c02d8 P feb95057296a4c968db65b492934213b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eca4f5b603bb4f87811018aa4ec07abb in term 0.
I20260620 18:32:31.473233 15030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56fd6eb46cf44f8eb11ed21a165c02d8" candidate_uuid: "eca4f5b603bb4f87811018aa4ec07abb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25051c5c5ac74f0e94343eb702029a06" is_pre_election: true
I20260620 18:32:31.473680 15030 raft_consensus.cc:2454] T 56fd6eb46cf44f8eb11ed21a165c02d8 P 25051c5c5ac74f0e94343eb702029a06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eca4f5b603bb4f87811018aa4ec07abb in term 0.
I20260620 18:32:31.473932 15096 leader_election.cc:302] T 56fd6eb46cf44f8eb11ed21a165c02d8 P eca4f5b603bb4f87811018aa4ec07abb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eca4f5b603bb4f87811018aa4ec07abb, feb95057296a4c968db65b492934213b; no voters:
I20260620 18:32:31.474279 15365 raft_consensus.cc:2781] T 56fd6eb46cf44f8eb11ed21a165c02d8 P eca4f5b603bb4f87811018aa4ec07abb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:32:31.474421 15365 raft_consensus.cc:482] T 56fd6eb46cf44f8eb11ed21a165c02d8 P eca4f5b603bb4f87811018aa4ec07abb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:32:31.474550 15365 raft_consensus.cc:3037] T 56fd6eb46cf44f8eb11ed21a165c02d8 P eca4f5b603bb4f87811018aa4ec07abb [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:31.476629 15365 raft_consensus.cc:504] T 56fd6eb46cf44f8eb11ed21a165c02d8 P eca4f5b603bb4f87811018aa4ec07abb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb95057296a4c968db65b492934213b" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40527 } } peers { permanent_uuid: "25051c5c5ac74f0e94343eb702029a06" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35929 } } peers { permanent_uuid: "eca4f5b603bb4f87811018aa4ec07abb" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39165 } }
I20260620 18:32:31.477262 15365 leader_election.cc:288] T 56fd6eb46cf44f8eb11ed21a165c02d8 P eca4f5b603bb4f87811018aa4ec07abb [CANDIDATE]: Term 1 election: Requested vote from peers feb95057296a4c968db65b492934213b (127.10.131.3:40527), 25051c5c5ac74f0e94343eb702029a06 (127.10.131.1:35929)
I20260620 18:32:31.477754 15030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56fd6eb46cf44f8eb11ed21a165c02d8" candidate_uuid: "eca4f5b603bb4f87811018aa4ec07abb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25051c5c5ac74f0e94343eb702029a06"
I20260620 18:32:31.477838 15295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56fd6eb46cf44f8eb11ed21a165c02d8" candidate_uuid: "eca4f5b603bb4f87811018aa4ec07abb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb95057296a4c968db65b492934213b"
I20260620 18:32:31.477952 15030 raft_consensus.cc:3037] T 56fd6eb46cf44f8eb11ed21a165c02d8 P 25051c5c5ac74f0e94343eb702029a06 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:31.478083 15295 raft_consensus.cc:3037] T 56fd6eb46cf44f8eb11ed21a165c02d8 P feb95057296a4c968db65b492934213b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:31.480288 15295 raft_consensus.cc:2454] T 56fd6eb46cf44f8eb11ed21a165c02d8 P feb95057296a4c968db65b492934213b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eca4f5b603bb4f87811018aa4ec07abb in term 1.
I20260620 18:32:31.480342 15030 raft_consensus.cc:2454] T 56fd6eb46cf44f8eb11ed21a165c02d8 P 25051c5c5ac74f0e94343eb702029a06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eca4f5b603bb4f87811018aa4ec07abb in term 1.
I20260620 18:32:31.480687 15096 leader_election.cc:302] T 56fd6eb46cf44f8eb11ed21a165c02d8 P eca4f5b603bb4f87811018aa4ec07abb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eca4f5b603bb4f87811018aa4ec07abb, feb95057296a4c968db65b492934213b; no voters:
I20260620 18:32:31.481007 15365 raft_consensus.cc:2781] T 56fd6eb46cf44f8eb11ed21a165c02d8 P eca4f5b603bb4f87811018aa4ec07abb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:31.481428 15365 raft_consensus.cc:686] T 56fd6eb46cf44f8eb11ed21a165c02d8 P eca4f5b603bb4f87811018aa4ec07abb [term 1 LEADER]: Becoming Leader. State: Replica: eca4f5b603bb4f87811018aa4ec07abb, State: Running, Role: LEADER
I20260620 18:32:31.481904 15365 consensus_queue.cc:237] T 56fd6eb46cf44f8eb11ed21a165c02d8 P eca4f5b603bb4f87811018aa4ec07abb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb95057296a4c968db65b492934213b" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40527 } } peers { permanent_uuid: "25051c5c5ac74f0e94343eb702029a06" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35929 } } peers { permanent_uuid: "eca4f5b603bb4f87811018aa4ec07abb" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39165 } }
I20260620 18:32:31.486377 14887 catalog_manager.cc:5495] T 56fd6eb46cf44f8eb11ed21a165c02d8 P eca4f5b603bb4f87811018aa4ec07abb reported cstate change: term changed from 0 to 1, leader changed from <none> to eca4f5b603bb4f87811018aa4ec07abb (127.10.131.2). New cstate: current_term: 1 leader_uuid: "eca4f5b603bb4f87811018aa4ec07abb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb95057296a4c968db65b492934213b" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25051c5c5ac74f0e94343eb702029a06" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eca4f5b603bb4f87811018aa4ec07abb" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39165 } health_report { overall_health: HEALTHY } } }
I20260620 18:32:31.517277 15357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:31.511945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58858 (local address 127.10.131.2:39165)
0620 18:32:31.512156 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:31.512164 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:31.512189 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:32:31.512450 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:31.512454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:31.512529 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:31.512684 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:32:31.513139 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.513849 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.515178 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.515577 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.515726 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:31.516193 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:32:31.516674 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:32:31.516866 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:31.516938 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 18:32:31.517013 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:32:31.525137 15030 raft_consensus.cc:1261] T 56fd6eb46cf44f8eb11ed21a165c02d8 P 25051c5c5ac74f0e94343eb702029a06 [term 1 FOLLOWER]: Refusing update from remote peer eca4f5b603bb4f87811018aa4ec07abb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:31.525133 15295 raft_consensus.cc:1261] T 56fd6eb46cf44f8eb11ed21a165c02d8 P feb95057296a4c968db65b492934213b [term 1 FOLLOWER]: Refusing update from remote peer eca4f5b603bb4f87811018aa4ec07abb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:31.526173 15369 consensus_queue.cc:1035] T 56fd6eb46cf44f8eb11ed21a165c02d8 P eca4f5b603bb4f87811018aa4ec07abb [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb95057296a4c968db65b492934213b" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:31.526522 15365 consensus_queue.cc:1035] T 56fd6eb46cf44f8eb11ed21a165c02d8 P eca4f5b603bb4f87811018aa4ec07abb [LEADER]: Connected to new peer: Peer: permanent_uuid: "25051c5c5ac74f0e94343eb702029a06" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:31.544013 15374 mvcc.cc:205] Tried to move back new op lower bound from 7298991519836753920 to 7298991519673458688. Current Snapshot: MvccSnapshot[applied={T|T < 7298991519836753920}]
I20260620 18:32:31.556618 14887 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:32:31.556915 14887 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:32:31.561257 14887 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 74a6055bde43485d98dd2e90ec1d0cc2: Sending DeleteTablet for 3 replicas of tablet 56fd6eb46cf44f8eb11ed21a165c02d8
I20260620 18:32:31.562198 15275 tablet_service.cc:1514] Processing DeleteTablet for tablet 56fd6eb46cf44f8eb11ed21a165c02d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:31 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:50820
I20260620 18:32:31.562438 15010 tablet_service.cc:1514] Processing DeleteTablet for tablet 56fd6eb46cf44f8eb11ed21a165c02d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:31 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:54406
I20260620 18:32:31.562444 15143 tablet_service.cc:1514] Processing DeleteTablet for tablet 56fd6eb46cf44f8eb11ed21a165c02d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:31 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:58854
I20260620 18:32:31.565871 15142 tablet_service.cc:1514] Processing DeleteTablet for tablet 56fd6eb46cf44f8eb11ed21a165c02d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:31 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:58854
I20260620 18:32:31.568081 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 14948
I20260620 18:32:31.575351 15382 tablet_replica.cc:330] T 56fd6eb46cf44f8eb11ed21a165c02d8 P feb95057296a4c968db65b492934213b: stopping tablet replica
I20260620 18:32:31.576656 15382 raft_consensus.cc:2229] T 56fd6eb46cf44f8eb11ed21a165c02d8 P feb95057296a4c968db65b492934213b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:32:31.577296 15382 raft_consensus.cc:2258] T 56fd6eb46cf44f8eb11ed21a165c02d8 P feb95057296a4c968db65b492934213b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:31.580744 15382 ts_tablet_manager.cc:1905] T 56fd6eb46cf44f8eb11ed21a165c02d8 P feb95057296a4c968db65b492934213b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:31.583113 15384 tablet_replica.cc:330] T 56fd6eb46cf44f8eb11ed21a165c02d8 P 25051c5c5ac74f0e94343eb702029a06: stopping tablet replica
I20260620 18:32:31.583595 15384 raft_consensus.cc:2229] T 56fd6eb46cf44f8eb11ed21a165c02d8 P 25051c5c5ac74f0e94343eb702029a06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:32:31.584215 15384 raft_consensus.cc:2258] T 56fd6eb46cf44f8eb11ed21a165c02d8 P 25051c5c5ac74f0e94343eb702029a06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:31.584965 15383 tablet_replica.cc:330] T 56fd6eb46cf44f8eb11ed21a165c02d8 P eca4f5b603bb4f87811018aa4ec07abb: stopping tablet replica
I20260620 18:32:31.585335 15383 raft_consensus.cc:2229] T 56fd6eb46cf44f8eb11ed21a165c02d8 P eca4f5b603bb4f87811018aa4ec07abb [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:32:31.586185 15383 raft_consensus.cc:2258] T 56fd6eb46cf44f8eb11ed21a165c02d8 P eca4f5b603bb4f87811018aa4ec07abb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:31.588289 15384 ts_tablet_manager.cc:1905] T 56fd6eb46cf44f8eb11ed21a165c02d8 P 25051c5c5ac74f0e94343eb702029a06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:31.588450 15383 ts_tablet_manager.cc:1905] T 56fd6eb46cf44f8eb11ed21a165c02d8 P eca4f5b603bb4f87811018aa4ec07abb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:31.592366 15382 ts_tablet_manager.cc:1918] T 56fd6eb46cf44f8eb11ed21a165c02d8 P feb95057296a4c968db65b492934213b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:32:31.592531 15382 log.cc:1196] T 56fd6eb46cf44f8eb11ed21a165c02d8 P feb95057296a4c968db65b492934213b: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-2/wal/wals/56fd6eb46cf44f8eb11ed21a165c02d8
I20260620 18:32:31.593143 15382 ts_tablet_manager.cc:1939] T 56fd6eb46cf44f8eb11ed21a165c02d8 P feb95057296a4c968db65b492934213b: Deleting consensus metadata
I20260620 18:32:31.594959 14871 catalog_manager.cc:4841] TS feb95057296a4c968db65b492934213b (127.10.131.3:40527): tablet 56fd6eb46cf44f8eb11ed21a165c02d8 (table test-table [id=c14711aba4974d76a3047e9d2382f9ce]) successfully deleted
I20260620 18:32:31.595417 15384 ts_tablet_manager.cc:1918] T 56fd6eb46cf44f8eb11ed21a165c02d8 P 25051c5c5ac74f0e94343eb702029a06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:32:31.595582 15384 log.cc:1196] T 56fd6eb46cf44f8eb11ed21a165c02d8 P 25051c5c5ac74f0e94343eb702029a06: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-0/wal/wals/56fd6eb46cf44f8eb11ed21a165c02d8
I20260620 18:32:31.596165 15384 ts_tablet_manager.cc:1939] T 56fd6eb46cf44f8eb11ed21a165c02d8 P 25051c5c5ac74f0e94343eb702029a06: Deleting consensus metadata
Jun 20 18:32:31 dist-test-slave-cbn3 krb5kdc[14843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.1@KRBTEST.COM
I20260620 18:32:31.597898 14872 catalog_manager.cc:4841] TS 25051c5c5ac74f0e94343eb702029a06 (127.10.131.1:35929): tablet 56fd6eb46cf44f8eb11ed21a165c02d8 (table test-table [id=c14711aba4974d76a3047e9d2382f9ce]) successfully deleted
I20260620 18:32:31.599248 15383 ts_tablet_manager.cc:1918] T 56fd6eb46cf44f8eb11ed21a165c02d8 P eca4f5b603bb4f87811018aa4ec07abb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:32:31.599428 15383 log.cc:1196] T 56fd6eb46cf44f8eb11ed21a165c02d8 P eca4f5b603bb4f87811018aa4ec07abb: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781980314095189-10764-0/minicluster-data/ts-1/wal/wals/56fd6eb46cf44f8eb11ed21a165c02d8
I20260620 18:32:31.600015 15383 ts_tablet_manager.cc:1939] T 56fd6eb46cf44f8eb11ed21a165c02d8 P eca4f5b603bb4f87811018aa4ec07abb: Deleting consensus metadata
I20260620 18:32:31.601564 14873 catalog_manager.cc:4841] TS eca4f5b603bb4f87811018aa4ec07abb (127.10.131.2:39165): tablet 56fd6eb46cf44f8eb11ed21a165c02d8 (table test-table [id=c14711aba4974d76a3047e9d2382f9ce]) successfully deleted
W20260620 18:32:31.602118 14873 catalog_manager.cc:4808] TS eca4f5b603bb4f87811018aa4ec07abb (127.10.131.2:39165): delete failed for tablet 56fd6eb46cf44f8eb11ed21a165c02d8 because the tablet was not found. No further retry: Not found: Tablet not found: 56fd6eb46cf44f8eb11ed21a165c02d8
I20260620 18:32:31.605670 15356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:31.569179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54422 (local address 127.10.131.1:35929)
0620 18:32:31.569419 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:31.569438 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:31.575557 (+ 6119us) server_negotiation.cc:407] Connection header received
0620 18:32:31.575794 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:31.575800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:31.575901 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:31.576004 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:31.583338 (+ 7334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.584530 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.589212 (+ 4682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.589865 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.589997 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:31.598734 (+ 8737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:31.598747 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:31.598765 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:31.598818 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:31.602639 (+ 3821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:31.603436 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:31.603445 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:31.603448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:31.603605 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:31.604198 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:31.604208 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:31.604211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:31.604619 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:31.605020 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:31.605237 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 18:32:31.605399 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 18:32:31.606576 15070 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:31.709043 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 14948
I20260620 18:32:31.731868 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 15081
Jun 20 18:32:31 dist-test-slave-cbn3 krb5kdc[14843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.2@KRBTEST.COM
I20260620 18:32:31.747452 15357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:31.732680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58872 (local address 127.10.131.2:39165)
0620 18:32:31.732876 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:31.732883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:31.732907 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:32:31.733149 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:31.733154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:31.733218 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:31.733343 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:31.734563 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.735385 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.736652 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.737302 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.737470 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:31.741664 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:31.741677 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:31.741695 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:31.741727 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:31.744699 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:31.745397 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:31.745405 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:31.745407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:31.745544 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:31.745923 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:31.745933 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:31.745935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:31.746264 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:31.746647 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:31.746854 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 18:32:31.747090 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:32:31.748162 15203 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:31.861970 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 15081
I20260620 18:32:31.880789 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 15213
Jun 20 18:32:31 dist-test-slave-cbn3 krb5kdc[14843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.3@KRBTEST.COM
I20260620 18:32:31.895836 15355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:31.881576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50828 (local address 127.10.131.3:40527)
0620 18:32:31.881772 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:31.881777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:31.881795 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:32:31.881909 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:31.881913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:31.881974 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:31.882086 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:31.883398 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.884165 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.885706 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:31.886132 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:31.886229 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:31.890587 (+ 4358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:31.890600 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:31.890616 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:31.890653 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:31.893467 (+ 2814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:31.894060 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:31.894067 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:31.894070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:31.894206 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:31.894512 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:31.894517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:31.894520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:31.894852 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:31.895095 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:31.895299 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 18:32:31.895457 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:32:31.896620 15335 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:31.985165 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 15213
I20260620 18:32:32.001534 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 14856
I20260620 18:32:32.002676 14917 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:32.110195 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 14856
2026-06-20T18:32:32Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4364 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T18:32:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:32:32Z Disabled control of system clock
I20260620 18:32:32.175771 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:42721
--webserver_interface=127.10.131.62
--webserver_port=0
--builtin_ntp_servers=127.10.131.20:33815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:42721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 18:32:32.491709 15393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:32.491983 15393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:32.492070 15393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:32.499308 15393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:32:32.499421 15393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:32.499459 15393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:32.499507 15393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:32:32.499550 15393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:32:32.508682 15393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:33815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:42721
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:42721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:32.509960 15393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:32.511714 15393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:32.523612 15398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:32.523717 15399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:32.523729 15401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:32.524794 15393 server_base.cc:1034] running on GCE node
I20260620 18:32:32.525661 15393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:32.527042 15393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:32.528301 15393 hybrid_clock.cc:648] HybridClock initialized: now 1781980352528255 us; error 35 us; skew 500 ppm
I20260620 18:32:32.528800 15393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:32:32.531085 15393 webserver.cc:460] Webserver started at http://127.10.131.62:33739/ using document root <none> and password file <none>
I20260620 18:32:32.531772 15393 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:32.531898 15393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:32.532217 15393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:32.535023 15393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "910c00ae863541a1a55ba926ca8884c8"
format_stamp: "Formatted at 2026-06-20 18:32:32 on dist-test-slave-cbn3"
server_key: "2dc8dde203882a6f493753d62bf2e417"
server_key_iv: "081a3050112a8cd957ba591e180eec44"
server_key_version: "encryptionkey@0"
I20260620 18:32:32.535840 15393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "910c00ae863541a1a55ba926ca8884c8"
format_stamp: "Formatted at 2026-06-20 18:32:32 on dist-test-slave-cbn3"
server_key: "2dc8dde203882a6f493753d62bf2e417"
server_key_iv: "081a3050112a8cd957ba591e180eec44"
server_key_version: "encryptionkey@0"
I20260620 18:32:32.540671 15393 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 18:32:32.543859 15407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:32.545095 15393 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 18:32:32.545259 15393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "910c00ae863541a1a55ba926ca8884c8"
format_stamp: "Formatted at 2026-06-20 18:32:32 on dist-test-slave-cbn3"
server_key: "2dc8dde203882a6f493753d62bf2e417"
server_key_iv: "081a3050112a8cd957ba591e180eec44"
server_key_version: "encryptionkey@0"
I20260620 18:32:32.545464 15393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:32.550872 15393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:32.551594 15393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:32.551884 15393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:32.569372 15393 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:42721
I20260620 18:32:32.569404 15458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:42721 every 8 connection(s)
I20260620 18:32:32.570647 15393 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:32:32.572522 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 15393
I20260620 18:32:32.572705 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:32:32.573556 10764 external_mini_cluster.cc:1428] Setting key 07e2f7c829a20045631d79fc01d8ce3d
I20260620 18:32:32.575070 15459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:32.581682 15462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:32.575732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43694 (local address 127.10.131.62:42721)
0620 18:32:32.576741 (+ 1009us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:32.576752 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:32.576790 (+ 38us) server_negotiation.cc:407] Connection header received
0620 18:32:32.578248 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:32.578266 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:32.578654 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:32.578955 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:32.579978 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:32.579987 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:32.579999 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:32:32.580012 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:32.580079 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:32:32.580098 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:32.580255 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:32.580368 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 18:32:32.580643 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":407,"threads_started":1}
I20260620 18:32:32.587879 15459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 910c00ae863541a1a55ba926ca8884c8: Bootstrap starting.
I20260620 18:32:32.590723 15459 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 910c00ae863541a1a55ba926ca8884c8: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:32.591905 15459 log.cc:824] T 00000000000000000000000000000000 P 910c00ae863541a1a55ba926ca8884c8: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:32.594408 15459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 910c00ae863541a1a55ba926ca8884c8: No bootstrap required, opened a new log
I20260620 18:32:32.601298 15459 raft_consensus.cc:348] T 00000000000000000000000000000000 P 910c00ae863541a1a55ba926ca8884c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910c00ae863541a1a55ba926ca8884c8" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 42721 } }
I20260620 18:32:32.601646 15459 raft_consensus.cc:374] T 00000000000000000000000000000000 P 910c00ae863541a1a55ba926ca8884c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:32.601755 15459 raft_consensus.cc:729] T 00000000000000000000000000000000 P 910c00ae863541a1a55ba926ca8884c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 910c00ae863541a1a55ba926ca8884c8, State: Initialized, Role: FOLLOWER
I20260620 18:32:32.602262 15459 consensus_queue.cc:260] T 00000000000000000000000000000000 P 910c00ae863541a1a55ba926ca8884c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910c00ae863541a1a55ba926ca8884c8" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 42721 } }
I20260620 18:32:32.602504 15459 raft_consensus.cc:388] T 00000000000000000000000000000000 P 910c00ae863541a1a55ba926ca8884c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:32:32.602638 15459 raft_consensus.cc:482] T 00000000000000000000000000000000 P 910c00ae863541a1a55ba926ca8884c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:32:32.602805 15459 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 910c00ae863541a1a55ba926ca8884c8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:32.605166 15459 raft_consensus.cc:504] T 00000000000000000000000000000000 P 910c00ae863541a1a55ba926ca8884c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910c00ae863541a1a55ba926ca8884c8" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 42721 } }
I20260620 18:32:32.605695 15459 leader_election.cc:302] T 00000000000000000000000000000000 P 910c00ae863541a1a55ba926ca8884c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 910c00ae863541a1a55ba926ca8884c8; no voters:
I20260620 18:32:32.606276 15459 leader_election.cc:288] T 00000000000000000000000000000000 P 910c00ae863541a1a55ba926ca8884c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:32:32.606424 15464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 910c00ae863541a1a55ba926ca8884c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:32.607213 15464 raft_consensus.cc:686] T 00000000000000000000000000000000 P 910c00ae863541a1a55ba926ca8884c8 [term 1 LEADER]: Becoming Leader. State: Replica: 910c00ae863541a1a55ba926ca8884c8, State: Running, Role: LEADER
I20260620 18:32:32.607826 15464 consensus_queue.cc:237] T 00000000000000000000000000000000 P 910c00ae863541a1a55ba926ca8884c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910c00ae863541a1a55ba926ca8884c8" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 42721 } }
I20260620 18:32:32.608430 15459 sys_catalog.cc:564] T 00000000000000000000000000000000 P 910c00ae863541a1a55ba926ca8884c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:32:32.610900 15465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 910c00ae863541a1a55ba926ca8884c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "910c00ae863541a1a55ba926ca8884c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910c00ae863541a1a55ba926ca8884c8" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 42721 } } }
I20260620 18:32:32.610924 15466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 910c00ae863541a1a55ba926ca8884c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 910c00ae863541a1a55ba926ca8884c8. Latest consensus state: current_term: 1 leader_uuid: "910c00ae863541a1a55ba926ca8884c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910c00ae863541a1a55ba926ca8884c8" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 42721 } } }
I20260620 18:32:32.611334 15465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 910c00ae863541a1a55ba926ca8884c8 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:32.611338 15466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 910c00ae863541a1a55ba926ca8884c8 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:32.615402 15471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:32:32.620151 15471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:32:32.629606 15471 catalog_manager.cc:1347] Generated new cluster ID: 3930f5d1d05b40c59df56e6a095cfa1a
I20260620 18:32:32.629731 15471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:32:32.641736 15471 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:32:32.642171 15471 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:32:32.648823 15471 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 910c00ae863541a1a55ba926ca8884c8: Generated new TSK 0
I20260620 18:32:32.649873 15471 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:32:32.662982 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.1:0
--local_ip_for_outbound_sockets=127.10.131.1
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:42721
--builtin_ntp_servers=127.10.131.20:33815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 18:32:33.016377 15483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:33.016724 15483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:33.016827 15483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:33.024384 15483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:33.024487 15483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:33.024602 15483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.1
I20260620 18:32:33.034044 15483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:33815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:42721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.1
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:33.035283 15483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:33.037155 15483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:33.050596 15489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:33.050523 15491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:33.050573 15488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:33.051759 15483 server_base.cc:1034] running on GCE node
I20260620 18:32:33.052335 15483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:33.053395 15483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:33.054589 15483 hybrid_clock.cc:648] HybridClock initialized: now 1781980353054553 us; error 25 us; skew 500 ppm
I20260620 18:32:33.054994 15483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:32:33.057408 15483 webserver.cc:460] Webserver started at http://127.10.131.1:32887/ using document root <none> and password file <none>
I20260620 18:32:33.058079 15483 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:33.058202 15483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:33.058532 15483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:33.061311 15483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-0/data/instance:
uuid: "a536c3d7cf4946fbb5277765ab3442c0"
format_stamp: "Formatted at 2026-06-20 18:32:33 on dist-test-slave-cbn3"
server_key: "17453be265eabe2d522c86a80f1ef391"
server_key_iv: "85b0f1d90f06797a9a20afbc11d8591e"
server_key_version: "encryptionkey@0"
I20260620 18:32:33.062085 15483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance:
uuid: "a536c3d7cf4946fbb5277765ab3442c0"
format_stamp: "Formatted at 2026-06-20 18:32:33 on dist-test-slave-cbn3"
server_key: "17453be265eabe2d522c86a80f1ef391"
server_key_iv: "85b0f1d90f06797a9a20afbc11d8591e"
server_key_version: "encryptionkey@0"
I20260620 18:32:33.067003 15483 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:32:33.070446 15497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:33.071956 15483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:32:33.072194 15483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-0/wal
uuid: "a536c3d7cf4946fbb5277765ab3442c0"
format_stamp: "Formatted at 2026-06-20 18:32:33 on dist-test-slave-cbn3"
server_key: "17453be265eabe2d522c86a80f1ef391"
server_key_iv: "85b0f1d90f06797a9a20afbc11d8591e"
server_key_version: "encryptionkey@0"
I20260620 18:32:33.072436 15483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:33.078346 15483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:33.079185 15483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:33.079486 15483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:33.080422 15483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:33.081949 15483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:33.082075 15483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:33.082254 15483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:33.082353 15483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:33.117311 15483 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.1:40753
I20260620 18:32:33.117331 15609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.1:40753 every 8 connection(s)
I20260620 18:32:33.118830 15483 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
I20260620 18:32:33.125372 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 15483
I20260620 18:32:33.125588 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance
I20260620 18:32:33.126425 10764 external_mini_cluster.cc:1428] Setting key 3d6f11c84fc094077806ac822534d9bb
I20260620 18:32:33.127058 15612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:33.121368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:39831 (local address 127.10.131.62:42721)
0620 18:32:33.121808 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:33.121812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:33.122857 (+ 1045us) server_negotiation.cc:407] Connection header received
0620 18:32:33.124477 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:33.124483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:33.124557 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:33.124672 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:33.126263 (+ 1591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:33.126268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:33.126272 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:32:33.126291 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:33.126337 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:32:33.126351 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:33.126460 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:33.126705 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 18:32:33.126758 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":234,"threads_started":1}
I20260620 18:32:33.128010 15613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:33.121864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:42721 (local address 127.10.131.1:39831)
0620 18:32:33.122706 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:33.122751 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:33.124227 (+ 1476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:33.124961 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:33.124972 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:33.125096 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:33.125903 (+ 807us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:32:33.125907 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:33.125958 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:32:33.126071 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:33.126462 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:33.126473 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:33.126482 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 18:32:33.126765 (+ 283us) client_negotiation.cc:240] Negotiation successful
0620 18:32:33.127004 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":255,"threads_started":1}
I20260620 18:32:33.129478 15610 heartbeater.cc:347] Connected to a master server at 127.10.131.62:42721
I20260620 18:32:33.129585 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.2:0
--local_ip_for_outbound_sockets=127.10.131.2
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:42721
--builtin_ntp_servers=127.10.131.20:33815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 18:32:33.129786 15610 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:33.130414 15610 heartbeater.cc:511] Master 127.10.131.62:42721 requested a full tablet report, sending...
I20260620 18:32:33.132452 15424 ts_manager.cc:194] Registered new tserver with Master: a536c3d7cf4946fbb5277765ab3442c0 (127.10.131.1:40753)
W20260620 18:32:33.460966 15614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:33.461275 15614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:33.461436 15614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:33.468596 15614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:33.468743 15614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:33.468961 15614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.2
I20260620 18:32:33.478291 15614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:33815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:42721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.2
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:33.479620 15614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:33.481444 15614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:33.494343 15619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:33.494303 15620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:33.495138 15622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:33.495990 15614 server_base.cc:1034] running on GCE node
I20260620 18:32:33.496778 15614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:33.498091 15614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:33.499353 15614 hybrid_clock.cc:648] HybridClock initialized: now 1781980353499298 us; error 55 us; skew 500 ppm
I20260620 18:32:33.499783 15614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:32:33.502403 15614 webserver.cc:460] Webserver started at http://127.10.131.2:37047/ using document root <none> and password file <none>
I20260620 18:32:33.503088 15614 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:33.503221 15614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:33.503554 15614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:33.506409 15614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-1/data/instance:
uuid: "2a77924848fd4c32b7aeb4729b79204d"
format_stamp: "Formatted at 2026-06-20 18:32:33 on dist-test-slave-cbn3"
server_key: "3e8ac74fdf00dd42cc8c5ec81ebdfd2d"
server_key_iv: "4c7bd0dcd951a6a5bba9b3cd7b435213"
server_key_version: "encryptionkey@0"
I20260620 18:32:33.507196 15614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance:
uuid: "2a77924848fd4c32b7aeb4729b79204d"
format_stamp: "Formatted at 2026-06-20 18:32:33 on dist-test-slave-cbn3"
server_key: "3e8ac74fdf00dd42cc8c5ec81ebdfd2d"
server_key_iv: "4c7bd0dcd951a6a5bba9b3cd7b435213"
server_key_version: "encryptionkey@0"
I20260620 18:32:33.512490 15614 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:32:33.515995 15628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:33.517549 15614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:32:33.517748 15614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-1/wal
uuid: "2a77924848fd4c32b7aeb4729b79204d"
format_stamp: "Formatted at 2026-06-20 18:32:33 on dist-test-slave-cbn3"
server_key: "3e8ac74fdf00dd42cc8c5ec81ebdfd2d"
server_key_iv: "4c7bd0dcd951a6a5bba9b3cd7b435213"
server_key_version: "encryptionkey@0"
I20260620 18:32:33.517982 15614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:33.524149 15614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:33.525072 15614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:33.525357 15614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:33.526314 15614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:33.527796 15614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:33.527930 15614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:33.528062 15614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:33.528155 15614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:33.562182 15614 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.2:37787
I20260620 18:32:33.562232 15740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.2:37787 every 8 connection(s)
I20260620 18:32:33.563481 15614 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
I20260620 18:32:33.564623 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 15614
I20260620 18:32:33.564805 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance
I20260620 18:32:33.565384 10764 external_mini_cluster.cc:1428] Setting key 14a0ed65f52af768e6a674e23497d707
I20260620 18:32:33.568383 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.3:0
--local_ip_for_outbound_sockets=127.10.131.3
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:42721
--builtin_ntp_servers=127.10.131.20:33815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 18:32:33.573236 15612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:33.566423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:40915 (local address 127.10.131.62:42721)
0620 18:32:33.566605 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:33.566611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:33.568091 (+ 1480us) server_negotiation.cc:407] Connection header received
0620 18:32:33.570824 (+ 2733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:33.570832 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:33.570899 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:33.571012 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:33.572510 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:33.572518 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:33.572526 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:32:33.572549 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:33.572591 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:32:33.572609 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:33.572738 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:33.572932 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 18:32:33.573003 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 18:32:33.574378 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:33.566961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:42721 (local address 127.10.131.2:40915)
0620 18:32:33.567944 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:33.567987 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:33.570511 (+ 2524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:33.571253 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:33.571264 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:33.571390 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:33.572145 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:32:33.572149 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:33.572199 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:32:33.572325 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:33.572712 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:33.572720 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:33.572728 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 18:32:33.572985 (+ 257us) client_negotiation.cc:240] Negotiation successful
0620 18:32:33.573218 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":352,"threads_started":1}
I20260620 18:32:33.575595 15741 heartbeater.cc:347] Connected to a master server at 127.10.131.62:42721
I20260620 18:32:33.575927 15741 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:33.576646 15741 heartbeater.cc:511] Master 127.10.131.62:42721 requested a full tablet report, sending...
I20260620 18:32:33.578147 15424 ts_manager.cc:194] Registered new tserver with Master: 2a77924848fd4c32b7aeb4729b79204d (127.10.131.2:37787)
W20260620 18:32:33.894986 15744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:33.895236 15744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:33.895321 15744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:33.902418 15744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:33.902513 15744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:33.902619 15744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.3
I20260620 18:32:33.911666 15744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:33815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:42721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.3
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:33.912881 15744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:33.914512 15744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:33.927670 15750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:33.927740 15749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:33.927776 15752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:33.928578 15744 server_base.cc:1034] running on GCE node
I20260620 18:32:33.929188 15744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:33.930119 15744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:33.931349 15744 hybrid_clock.cc:648] HybridClock initialized: now 1781980353931309 us; error 53 us; skew 500 ppm
I20260620 18:32:33.931660 15744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:32:33.933946 15744 webserver.cc:460] Webserver started at http://127.10.131.3:45393/ using document root <none> and password file <none>
I20260620 18:32:33.934598 15744 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:33.934727 15744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:33.935057 15744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:33.937844 15744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-2/data/instance:
uuid: "4ecc8c3409134198b74ab97f3f73384f"
format_stamp: "Formatted at 2026-06-20 18:32:33 on dist-test-slave-cbn3"
server_key: "f0ac39f0eecea741f29ce1785d3a78ad"
server_key_iv: "aeb0390726aa7aca5a7a5e345cc87b6e"
server_key_version: "encryptionkey@0"
I20260620 18:32:33.938623 15744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance:
uuid: "4ecc8c3409134198b74ab97f3f73384f"
format_stamp: "Formatted at 2026-06-20 18:32:33 on dist-test-slave-cbn3"
server_key: "f0ac39f0eecea741f29ce1785d3a78ad"
server_key_iv: "aeb0390726aa7aca5a7a5e345cc87b6e"
server_key_version: "encryptionkey@0"
I20260620 18:32:33.943476 15744 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 18:32:33.946762 15758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:33.947968 15744 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 18:32:33.948145 15744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-2/wal
uuid: "4ecc8c3409134198b74ab97f3f73384f"
format_stamp: "Formatted at 2026-06-20 18:32:33 on dist-test-slave-cbn3"
server_key: "f0ac39f0eecea741f29ce1785d3a78ad"
server_key_iv: "aeb0390726aa7aca5a7a5e345cc87b6e"
server_key_version: "encryptionkey@0"
I20260620 18:32:33.948412 15744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:33.953948 15744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:33.954690 15744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:33.954962 15744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:33.955811 15744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:33.957196 15744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:33.957327 15744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:33.957463 15744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:33.957551 15744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:33.989679 15744 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.3:41161
I20260620 18:32:33.989728 15870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.3:41161 every 8 connection(s)
I20260620 18:32:33.991009 15744 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
I20260620 18:32:33.992148 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 15744
I20260620 18:32:33.992307 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance
I20260620 18:32:33.992889 10764 external_mini_cluster.cc:1428] Setting key da8613dac4e48d6bd8b6cb5277105287
I20260620 18:32:33.999473 15612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:33.993810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:44867 (local address 127.10.131.62:42721)
0620 18:32:33.993983 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:33.993987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:33.995673 (+ 1686us) server_negotiation.cc:407] Connection header received
0620 18:32:33.997500 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:33.997504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:33.997555 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:33.997637 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:33.998874 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:33.998878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:33.998881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:32:33.998897 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:33.998928 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:32:33.998940 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:33.999045 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:33.999199 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 18:32:33.999248 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:32:34.000315 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:33.994474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:42721 (local address 127.10.131.3:44867)
0620 18:32:33.995537 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:33.995584 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:33.997261 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:33.997830 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:33.997838 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:33.997943 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:33.998587 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:32:33.998589 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:33.998624 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:32:33.998705 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:33.999029 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:33.999035 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:33.999040 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:32:33.999242 (+ 202us) client_negotiation.cc:240] Negotiation successful
0620 18:32:33.999428 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":300,"threads_started":1}
I20260620 18:32:34.001590 15871 heartbeater.cc:347] Connected to a master server at 127.10.131.62:42721
I20260620 18:32:34.001876 15871 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:34.002437 15871 heartbeater.cc:511] Master 127.10.131.62:42721 requested a full tablet report, sending...
I20260620 18:32:34.003726 15424 ts_manager.cc:194] Registered new tserver with Master: 4ecc8c3409134198b74ab97f3f73384f (127.10.131.3:41161)
I20260620 18:32:34.008165 10764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:32:34.015203 15612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:34.012458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43704 (local address 127.10.131.62:42721)
0620 18:32:34.012642 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:34.012650 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:34.013347 (+ 697us) server_negotiation.cc:407] Connection header received
0620 18:32:34.013502 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:34.013506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:34.013557 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:34.013634 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:34.014626 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:34.014629 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:34.014632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:32:34.014646 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:34.014672 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:32:34.014682 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:34.014759 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:34.014849 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 18:32:34.014896 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 18:32:34.019769 15424 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:32:34.026736 15424 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:32:34.057404 15887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:34.038274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:37787 (local address 127.0.0.1:51678)
0620 18:32:34.049991 (+ 11717us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:34.050054 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:34.050268 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:34.053201 (+ 2933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:34.053213 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:34.053357 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:34.054659 (+ 1302us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:32:34.054664 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:34.054700 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:32:34.054786 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:34.055437 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:34.055447 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:34.055457 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 18:32:34.055639 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 18:32:34.055729 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11353,"thread_start_us":16702,"threads_started":1}
I20260620 18:32:34.059173 15885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:34.038514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51678 (local address 127.10.131.2:37787)
0620 18:32:34.051210 (+ 12696us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:34.051223 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:34.051277 (+ 54us) server_negotiation.cc:407] Connection header received
0620 18:32:34.051375 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:34.051388 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:34.051797 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:34.052006 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:34.055009 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:34.055019 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:34.055034 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:32:34.055065 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:34.055124 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:32:34.055148 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:34.055381 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:34.058561 (+ 3180us) server_negotiation.cc:299] Negotiation successful
0620 18:32:34.058643 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12467,"thread_start_us":12296,"threads_started":1}
I20260620 18:32:34.059201 15886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:34.038756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59174 (local address 127.10.131.3:41161)
0620 18:32:34.049183 (+ 10427us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:34.049198 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:34.054958 (+ 5760us) server_negotiation.cc:407] Connection header received
0620 18:32:34.055602 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:34.055614 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:34.055837 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:34.055972 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:34.058193 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:34.058202 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:34.058216 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:32:34.058240 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:34.058311 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:32:34.058335 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:34.058486 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:34.058747 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 18:32:34.058833 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10165,"thread_start_us":11655,"threads_started":1}
I20260620 18:32:34.059266 15883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:34.038412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:41161 (local address 127.0.0.1:59174)
0620 18:32:34.054836 (+ 16424us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:34.054863 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:34.055428 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:34.057095 (+ 1667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:34.057100 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:34.057123 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:34.057934 (+ 811us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:32:34.057940 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:34.057961 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:32:34.058014 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:34.058586 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:34.058590 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:34.058595 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:32:34.058738 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 18:32:34.058802 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16267,"thread_start_us":19859,"threads_started":1}
I20260620 18:32:34.061651 15887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:34.040467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:40753 (local address 127.0.0.1:37718)
0620 18:32:34.058215 (+ 17748us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:34.058239 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:34.058349 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:34.059937 (+ 1588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:34.059942 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:34.059982 (+ 40us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:34.060531 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:32:34.060535 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:34.060550 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:32:34.060593 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:34.061274 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:34.061279 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:34.061283 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 18:32:34.061371 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 18:32:34.061420 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17645,"thread_start_us":8959,"threads_started":1}
I20260620 18:32:34.061798 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:34.041120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37718 (local address 127.10.131.1:40753)
0620 18:32:34.057876 (+ 16756us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:34.057888 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:34.058538 (+ 650us) server_negotiation.cc:407] Connection header received
0620 18:32:34.058627 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:34.058637 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:34.058853 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:34.060233 (+ 1380us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:34.061102 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:34.061107 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:34.061116 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:32:34.061132 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:34.061164 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:32:34.061176 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:34.061314 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:34.061400 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 18:32:34.061496 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16543,"thread_start_us":16761,"threads_started":1}
I20260620 18:32:34.063761 15806 tablet_service.cc:1467] Processing CreateTablet for tablet aa204ad0d4d947018c4c191e8d05087f (DEFAULT_TABLE table=test-table [id=5d9fc4e1b3e146638ca2654857a59b28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:34.064512 15545 tablet_service.cc:1467] Processing CreateTablet for tablet aa204ad0d4d947018c4c191e8d05087f (DEFAULT_TABLE table=test-table [id=5d9fc4e1b3e146638ca2654857a59b28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:34.064661 15676 tablet_service.cc:1467] Processing CreateTablet for tablet aa204ad0d4d947018c4c191e8d05087f (DEFAULT_TABLE table=test-table [id=5d9fc4e1b3e146638ca2654857a59b28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:34.065382 15806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa204ad0d4d947018c4c191e8d05087f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:34.065811 15545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa204ad0d4d947018c4c191e8d05087f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:34.065949 15676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa204ad0d4d947018c4c191e8d05087f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:34.077266 15889 tablet_bootstrap.cc:492] T aa204ad0d4d947018c4c191e8d05087f P a536c3d7cf4946fbb5277765ab3442c0: Bootstrap starting.
I20260620 18:32:34.079815 15890 tablet_bootstrap.cc:492] T aa204ad0d4d947018c4c191e8d05087f P 2a77924848fd4c32b7aeb4729b79204d: Bootstrap starting.
I20260620 18:32:34.081674 15889 tablet_bootstrap.cc:654] T aa204ad0d4d947018c4c191e8d05087f P a536c3d7cf4946fbb5277765ab3442c0: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:34.081882 15891 tablet_bootstrap.cc:492] T aa204ad0d4d947018c4c191e8d05087f P 4ecc8c3409134198b74ab97f3f73384f: Bootstrap starting.
I20260620 18:32:34.082983 15889 log.cc:824] T aa204ad0d4d947018c4c191e8d05087f P a536c3d7cf4946fbb5277765ab3442c0: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:34.083345 15890 tablet_bootstrap.cc:654] T aa204ad0d4d947018c4c191e8d05087f P 2a77924848fd4c32b7aeb4729b79204d: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:34.084775 15890 log.cc:824] T aa204ad0d4d947018c4c191e8d05087f P 2a77924848fd4c32b7aeb4729b79204d: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:34.084965 15891 tablet_bootstrap.cc:654] T aa204ad0d4d947018c4c191e8d05087f P 4ecc8c3409134198b74ab97f3f73384f: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:34.086170 15891 log.cc:824] T aa204ad0d4d947018c4c191e8d05087f P 4ecc8c3409134198b74ab97f3f73384f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:34.087021 15889 tablet_bootstrap.cc:492] T aa204ad0d4d947018c4c191e8d05087f P a536c3d7cf4946fbb5277765ab3442c0: No bootstrap required, opened a new log
I20260620 18:32:34.087327 15889 ts_tablet_manager.cc:1397] T aa204ad0d4d947018c4c191e8d05087f P a536c3d7cf4946fbb5277765ab3442c0: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260620 18:32:34.087858 15890 tablet_bootstrap.cc:492] T aa204ad0d4d947018c4c191e8d05087f P 2a77924848fd4c32b7aeb4729b79204d: No bootstrap required, opened a new log
I20260620 18:32:34.088097 15890 ts_tablet_manager.cc:1397] T aa204ad0d4d947018c4c191e8d05087f P 2a77924848fd4c32b7aeb4729b79204d: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260620 18:32:34.088984 15891 tablet_bootstrap.cc:492] T aa204ad0d4d947018c4c191e8d05087f P 4ecc8c3409134198b74ab97f3f73384f: No bootstrap required, opened a new log
I20260620 18:32:34.089216 15891 ts_tablet_manager.cc:1397] T aa204ad0d4d947018c4c191e8d05087f P 4ecc8c3409134198b74ab97f3f73384f: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260620 18:32:34.095809 15891 raft_consensus.cc:348] T aa204ad0d4d947018c4c191e8d05087f P 4ecc8c3409134198b74ab97f3f73384f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a77924848fd4c32b7aeb4729b79204d" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 37787 } } peers { permanent_uuid: "4ecc8c3409134198b74ab97f3f73384f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 41161 } } peers { permanent_uuid: "a536c3d7cf4946fbb5277765ab3442c0" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40753 } }
I20260620 18:32:34.096408 15891 raft_consensus.cc:374] T aa204ad0d4d947018c4c191e8d05087f P 4ecc8c3409134198b74ab97f3f73384f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:34.096565 15891 raft_consensus.cc:729] T aa204ad0d4d947018c4c191e8d05087f P 4ecc8c3409134198b74ab97f3f73384f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecc8c3409134198b74ab97f3f73384f, State: Initialized, Role: FOLLOWER
I20260620 18:32:34.097254 15891 consensus_queue.cc:260] T aa204ad0d4d947018c4c191e8d05087f P 4ecc8c3409134198b74ab97f3f73384f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a77924848fd4c32b7aeb4729b79204d" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 37787 } } peers { permanent_uuid: "4ecc8c3409134198b74ab97f3f73384f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 41161 } } peers { permanent_uuid: "a536c3d7cf4946fbb5277765ab3442c0" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40753 } }
I20260620 18:32:34.098373 15871 heartbeater.cc:503] Master 127.10.131.62:42721 was elected leader, sending a full tablet report...
I20260620 18:32:34.098560 15890 raft_consensus.cc:348] T aa204ad0d4d947018c4c191e8d05087f P 2a77924848fd4c32b7aeb4729b79204d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a77924848fd4c32b7aeb4729b79204d" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 37787 } } peers { permanent_uuid: "4ecc8c3409134198b74ab97f3f73384f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 41161 } } peers { permanent_uuid: "a536c3d7cf4946fbb5277765ab3442c0" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40753 } }
I20260620 18:32:34.098560 15889 raft_consensus.cc:348] T aa204ad0d4d947018c4c191e8d05087f P a536c3d7cf4946fbb5277765ab3442c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a77924848fd4c32b7aeb4729b79204d" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 37787 } } peers { permanent_uuid: "4ecc8c3409134198b74ab97f3f73384f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 41161 } } peers { permanent_uuid: "a536c3d7cf4946fbb5277765ab3442c0" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40753 } }
I20260620 18:32:34.098991 15890 raft_consensus.cc:374] T aa204ad0d4d947018c4c191e8d05087f P 2a77924848fd4c32b7aeb4729b79204d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:34.098991 15889 raft_consensus.cc:374] T aa204ad0d4d947018c4c191e8d05087f P a536c3d7cf4946fbb5277765ab3442c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:34.099068 15891 ts_tablet_manager.cc:1428] T aa204ad0d4d947018c4c191e8d05087f P 4ecc8c3409134198b74ab97f3f73384f: Time spent starting tablet: real 0.010s user 0.006s sys 0.003s
I20260620 18:32:34.099081 15890 raft_consensus.cc:729] T aa204ad0d4d947018c4c191e8d05087f P 2a77924848fd4c32b7aeb4729b79204d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a77924848fd4c32b7aeb4729b79204d, State: Initialized, Role: FOLLOWER
I20260620 18:32:34.099081 15889 raft_consensus.cc:729] T aa204ad0d4d947018c4c191e8d05087f P a536c3d7cf4946fbb5277765ab3442c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a536c3d7cf4946fbb5277765ab3442c0, State: Initialized, Role: FOLLOWER
I20260620 18:32:34.099659 15889 consensus_queue.cc:260] T aa204ad0d4d947018c4c191e8d05087f P a536c3d7cf4946fbb5277765ab3442c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a77924848fd4c32b7aeb4729b79204d" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 37787 } } peers { permanent_uuid: "4ecc8c3409134198b74ab97f3f73384f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 41161 } } peers { permanent_uuid: "a536c3d7cf4946fbb5277765ab3442c0" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40753 } }
I20260620 18:32:34.099854 15890 consensus_queue.cc:260] T aa204ad0d4d947018c4c191e8d05087f P 2a77924848fd4c32b7aeb4729b79204d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a77924848fd4c32b7aeb4729b79204d" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 37787 } } peers { permanent_uuid: "4ecc8c3409134198b74ab97f3f73384f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 41161 } } peers { permanent_uuid: "a536c3d7cf4946fbb5277765ab3442c0" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40753 } }
I20260620 18:32:34.100917 15741 heartbeater.cc:503] Master 127.10.131.62:42721 was elected leader, sending a full tablet report...
I20260620 18:32:34.101795 15890 ts_tablet_manager.cc:1428] T aa204ad0d4d947018c4c191e8d05087f P 2a77924848fd4c32b7aeb4729b79204d: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260620 18:32:34.102690 15610 heartbeater.cc:503] Master 127.10.131.62:42721 was elected leader, sending a full tablet report...
I20260620 18:32:34.103227 15889 ts_tablet_manager.cc:1428] T aa204ad0d4d947018c4c191e8d05087f P a536c3d7cf4946fbb5277765ab3442c0: Time spent starting tablet: real 0.016s user 0.013s sys 0.001s
W20260620 18:32:34.121109 15611 tablet.cc:2369] T aa204ad0d4d947018c4c191e8d05087f P a536c3d7cf4946fbb5277765ab3442c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:32:34.164755 15895 raft_consensus.cc:482] T aa204ad0d4d947018c4c191e8d05087f P 4ecc8c3409134198b74ab97f3f73384f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:32:34.165118 15895 raft_consensus.cc:504] T aa204ad0d4d947018c4c191e8d05087f P 4ecc8c3409134198b74ab97f3f73384f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a77924848fd4c32b7aeb4729b79204d" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 37787 } } peers { permanent_uuid: "4ecc8c3409134198b74ab97f3f73384f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 41161 } } peers { permanent_uuid: "a536c3d7cf4946fbb5277765ab3442c0" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40753 } }
I20260620 18:32:34.166818 15895 leader_election.cc:288] T aa204ad0d4d947018c4c191e8d05087f P 4ecc8c3409134198b74ab97f3f73384f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a77924848fd4c32b7aeb4729b79204d (127.10.131.2:37787), a536c3d7cf4946fbb5277765ab3442c0 (127.10.131.1:40753)
I20260620 18:32:34.169675 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:34.167404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:44979 (local address 127.10.131.1:40753)
0620 18:32:34.167602 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:34.167607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:34.167920 (+ 313us) server_negotiation.cc:407] Connection header received
0620 18:32:34.168027 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:34.168034 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:34.168097 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:34.168210 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:34.169036 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:34.169043 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:34.169047 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:32:34.169068 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:34.169102 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:32:34.169117 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:34.169240 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:34.169328 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 18:32:34.169389 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 18:32:34.169677 15885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:34.167359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:51395 (local address 127.10.131.2:37787)
0620 18:32:34.167571 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:34.167578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:34.167604 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:32:34.167760 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:34.167776 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:34.167847 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:34.167985 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:34.169019 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:34.169026 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:34.169030 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:32:34.169050 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:34.169095 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:32:34.169112 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:34.169249 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:34.169328 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 18:32:34.169389 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:32:34.169682 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:34.167151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:37787 (local address 127.10.131.3:51395)
0620 18:32:34.167463 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:34.167488 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:34.167606 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:34.168030 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:34.168033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:34.168051 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:34.168722 (+ 671us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:32:34.168727 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:34.168747 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:32:34.168802 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:34.169220 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:34.169225 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:34.169230 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:32:34.169332 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 18:32:34.169395 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182}
I20260620 18:32:34.169837 15898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:34.167211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:40753 (local address 127.10.131.3:44979)
0620 18:32:34.167818 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:34.167831 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:34.167901 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:34.168263 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:34.168269 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:34.168312 (+ 43us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:34.168803 (+ 491us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:32:34.168806 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:34.168816 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:32:34.168884 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:34.169220 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:34.169225 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:34.169230 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:32:34.169339 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 18:32:34.169406 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":21,"thread_start_us":365,"threads_started":1}
I20260620 18:32:34.170666 15696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa204ad0d4d947018c4c191e8d05087f" candidate_uuid: "4ecc8c3409134198b74ab97f3f73384f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a77924848fd4c32b7aeb4729b79204d" is_pre_election: true
I20260620 18:32:34.170869 15565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa204ad0d4d947018c4c191e8d05087f" candidate_uuid: "4ecc8c3409134198b74ab97f3f73384f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a536c3d7cf4946fbb5277765ab3442c0" is_pre_election: true
I20260620 18:32:34.171169 15696 raft_consensus.cc:2454] T aa204ad0d4d947018c4c191e8d05087f P 2a77924848fd4c32b7aeb4729b79204d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecc8c3409134198b74ab97f3f73384f in term 0.
I20260620 18:32:34.171449 15565 raft_consensus.cc:2454] T aa204ad0d4d947018c4c191e8d05087f P a536c3d7cf4946fbb5277765ab3442c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecc8c3409134198b74ab97f3f73384f in term 0.
I20260620 18:32:34.171953 15762 leader_election.cc:302] T aa204ad0d4d947018c4c191e8d05087f P 4ecc8c3409134198b74ab97f3f73384f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a77924848fd4c32b7aeb4729b79204d, 4ecc8c3409134198b74ab97f3f73384f; no voters:
I20260620 18:32:34.172340 15895 raft_consensus.cc:2781] T aa204ad0d4d947018c4c191e8d05087f P 4ecc8c3409134198b74ab97f3f73384f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:32:34.172484 15895 raft_consensus.cc:482] T aa204ad0d4d947018c4c191e8d05087f P 4ecc8c3409134198b74ab97f3f73384f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:32:34.172585 15895 raft_consensus.cc:3037] T aa204ad0d4d947018c4c191e8d05087f P 4ecc8c3409134198b74ab97f3f73384f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:34.174722 15895 raft_consensus.cc:504] T aa204ad0d4d947018c4c191e8d05087f P 4ecc8c3409134198b74ab97f3f73384f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a77924848fd4c32b7aeb4729b79204d" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 37787 } } peers { permanent_uuid: "4ecc8c3409134198b74ab97f3f73384f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 41161 } } peers { permanent_uuid: "a536c3d7cf4946fbb5277765ab3442c0" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40753 } }
I20260620 18:32:34.175300 15895 leader_election.cc:288] T aa204ad0d4d947018c4c191e8d05087f P 4ecc8c3409134198b74ab97f3f73384f [CANDIDATE]: Term 1 election: Requested vote from peers 2a77924848fd4c32b7aeb4729b79204d (127.10.131.2:37787), a536c3d7cf4946fbb5277765ab3442c0 (127.10.131.1:40753)
I20260620 18:32:34.175705 15696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa204ad0d4d947018c4c191e8d05087f" candidate_uuid: "4ecc8c3409134198b74ab97f3f73384f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a77924848fd4c32b7aeb4729b79204d"
I20260620 18:32:34.175843 15565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa204ad0d4d947018c4c191e8d05087f" candidate_uuid: "4ecc8c3409134198b74ab97f3f73384f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a536c3d7cf4946fbb5277765ab3442c0"
I20260620 18:32:34.175908 15696 raft_consensus.cc:3037] T aa204ad0d4d947018c4c191e8d05087f P 2a77924848fd4c32b7aeb4729b79204d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:34.176103 15565 raft_consensus.cc:3037] T aa204ad0d4d947018c4c191e8d05087f P a536c3d7cf4946fbb5277765ab3442c0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:34.178328 15696 raft_consensus.cc:2454] T aa204ad0d4d947018c4c191e8d05087f P 2a77924848fd4c32b7aeb4729b79204d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecc8c3409134198b74ab97f3f73384f in term 1.
I20260620 18:32:34.178354 15565 raft_consensus.cc:2454] T aa204ad0d4d947018c4c191e8d05087f P a536c3d7cf4946fbb5277765ab3442c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecc8c3409134198b74ab97f3f73384f in term 1.
I20260620 18:32:34.178676 15762 leader_election.cc:302] T aa204ad0d4d947018c4c191e8d05087f P 4ecc8c3409134198b74ab97f3f73384f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a77924848fd4c32b7aeb4729b79204d, 4ecc8c3409134198b74ab97f3f73384f; no voters:
I20260620 18:32:34.178977 15895 raft_consensus.cc:2781] T aa204ad0d4d947018c4c191e8d05087f P 4ecc8c3409134198b74ab97f3f73384f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:34.179458 15895 raft_consensus.cc:686] T aa204ad0d4d947018c4c191e8d05087f P 4ecc8c3409134198b74ab97f3f73384f [term 1 LEADER]: Becoming Leader. State: Replica: 4ecc8c3409134198b74ab97f3f73384f, State: Running, Role: LEADER
I20260620 18:32:34.180002 15895 consensus_queue.cc:237] T aa204ad0d4d947018c4c191e8d05087f P 4ecc8c3409134198b74ab97f3f73384f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a77924848fd4c32b7aeb4729b79204d" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 37787 } } peers { permanent_uuid: "4ecc8c3409134198b74ab97f3f73384f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 41161 } } peers { permanent_uuid: "a536c3d7cf4946fbb5277765ab3442c0" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40753 } }
I20260620 18:32:34.184989 15423 catalog_manager.cc:5495] T aa204ad0d4d947018c4c191e8d05087f P 4ecc8c3409134198b74ab97f3f73384f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ecc8c3409134198b74ab97f3f73384f (127.10.131.3). New cstate: current_term: 1 leader_uuid: "4ecc8c3409134198b74ab97f3f73384f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a77924848fd4c32b7aeb4729b79204d" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 37787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ecc8c3409134198b74ab97f3f73384f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 41161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a536c3d7cf4946fbb5277765ab3442c0" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40753 } health_report { overall_health: UNKNOWN } } }
I20260620 18:32:34.219414 15886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:34.217577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59190 (local address 127.10.131.3:41161)
0620 18:32:34.217748 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:34.217753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:34.217802 (+ 49us) server_negotiation.cc:407] Connection header received
0620 18:32:34.217994 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:34.217998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:34.218076 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:34.218159 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:34.218933 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:34.218937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:34.218940 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:32:34.218955 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:34.218989 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:32:34.219000 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:34.219106 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:34.219167 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 18:32:34.219210 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:32:34.227528 15696 raft_consensus.cc:1261] T aa204ad0d4d947018c4c191e8d05087f P 2a77924848fd4c32b7aeb4729b79204d [term 1 FOLLOWER]: Refusing update from remote peer 4ecc8c3409134198b74ab97f3f73384f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:34.227602 15565 raft_consensus.cc:1261] T aa204ad0d4d947018c4c191e8d05087f P a536c3d7cf4946fbb5277765ab3442c0 [term 1 FOLLOWER]: Refusing update from remote peer 4ecc8c3409134198b74ab97f3f73384f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:34.228469 15899 consensus_queue.cc:1035] T aa204ad0d4d947018c4c191e8d05087f P 4ecc8c3409134198b74ab97f3f73384f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a77924848fd4c32b7aeb4729b79204d" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 37787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:34.228811 15895 consensus_queue.cc:1035] T aa204ad0d4d947018c4c191e8d05087f P 4ecc8c3409134198b74ab97f3f73384f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a536c3d7cf4946fbb5277765ab3442c0" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:34.241235 15903 mvcc.cc:205] Tried to move back new op lower bound from 7298991530906521600 to 7298991530725666816. Current Snapshot: MvccSnapshot[applied={T|T < 7298991530906521600}]
I20260620 18:32:34.244468 15902 mvcc.cc:205] Tried to move back new op lower bound from 7298991530906521600 to 7298991530725666816. Current Snapshot: MvccSnapshot[applied={T|T < 7298991530906521600 or (T in {7298991530906521600})}]
I20260620 18:32:34.251066 15904 mvcc.cc:205] Tried to move back new op lower bound from 7298991530906521600 to 7298991530725666816. Current Snapshot: MvccSnapshot[applied={T|T < 7298991530906521600}]
I20260620 18:32:34.254558 15422 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:32:34.254895 15422 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:32:34.258668 15422 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 910c00ae863541a1a55ba926ca8884c8: Sending DeleteTablet for 3 replicas of tablet aa204ad0d4d947018c4c191e8d05087f
I20260620 18:32:34.259716 15676 tablet_service.cc:1514] Processing DeleteTablet for tablet aa204ad0d4d947018c4c191e8d05087f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:34 UTC) from {username='slave'} at 127.0.0.1:51678
I20260620 18:32:34.259742 15806 tablet_service.cc:1514] Processing DeleteTablet for tablet aa204ad0d4d947018c4c191e8d05087f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:34 UTC) from {username='slave'} at 127.0.0.1:59174
I20260620 18:32:34.260591 15912 tablet_replica.cc:330] T aa204ad0d4d947018c4c191e8d05087f P 2a77924848fd4c32b7aeb4729b79204d: stopping tablet replica
I20260620 18:32:34.260634 15545 tablet_service.cc:1514] Processing DeleteTablet for tablet aa204ad0d4d947018c4c191e8d05087f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:34 UTC) from {username='slave'} at 127.0.0.1:37718
I20260620 18:32:34.261034 15912 raft_consensus.cc:2229] T aa204ad0d4d947018c4c191e8d05087f P 2a77924848fd4c32b7aeb4729b79204d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:32:34.261543 15912 raft_consensus.cc:2258] T aa204ad0d4d947018c4c191e8d05087f P 2a77924848fd4c32b7aeb4729b79204d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:34.262817 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 15483
I20260620 18:32:34.267678 15912 ts_tablet_manager.cc:1905] T aa204ad0d4d947018c4c191e8d05087f P 2a77924848fd4c32b7aeb4729b79204d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:34.271174 15914 tablet_replica.cc:330] T aa204ad0d4d947018c4c191e8d05087f P a536c3d7cf4946fbb5277765ab3442c0: stopping tablet replica
I20260620 18:32:34.271692 15914 raft_consensus.cc:2229] T aa204ad0d4d947018c4c191e8d05087f P a536c3d7cf4946fbb5277765ab3442c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:32:34.272248 15914 raft_consensus.cc:2258] T aa204ad0d4d947018c4c191e8d05087f P a536c3d7cf4946fbb5277765ab3442c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:34.274902 15914 ts_tablet_manager.cc:1905] T aa204ad0d4d947018c4c191e8d05087f P a536c3d7cf4946fbb5277765ab3442c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:34.278045 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:34.264483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37730 (local address 127.10.131.1:40753)
0620 18:32:34.264692 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:34.264701 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:34.275396 (+ 10695us) server_negotiation.cc:407] Connection header received
0620 18:32:34.275756 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:34.275763 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:34.275845 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:34.275935 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:34.277257 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:34.277263 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:34.277267 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:32:34.277286 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:34.277320 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:32:34.277334 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:34.277425 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:34.277673 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 18:32:34.277731 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:32:34.279196 15605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:34.280140 15913 tablet_replica.cc:330] T aa204ad0d4d947018c4c191e8d05087f P 4ecc8c3409134198b74ab97f3f73384f: stopping tablet replica
I20260620 18:32:34.280745 15913 raft_consensus.cc:2229] T aa204ad0d4d947018c4c191e8d05087f P 4ecc8c3409134198b74ab97f3f73384f [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:32:34.281268 15912 ts_tablet_manager.cc:1918] T aa204ad0d4d947018c4c191e8d05087f P 2a77924848fd4c32b7aeb4729b79204d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:32:34.281507 15912 log.cc:1196] T aa204ad0d4d947018c4c191e8d05087f P 2a77924848fd4c32b7aeb4729b79204d: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-1/wal/wals/aa204ad0d4d947018c4c191e8d05087f
I20260620 18:32:34.281670 15913 raft_consensus.cc:2258] T aa204ad0d4d947018c4c191e8d05087f P 4ecc8c3409134198b74ab97f3f73384f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:34.282400 15912 ts_tablet_manager.cc:1939] T aa204ad0d4d947018c4c191e8d05087f P 2a77924848fd4c32b7aeb4729b79204d: Deleting consensus metadata
I20260620 18:32:34.284356 15913 ts_tablet_manager.cc:1905] T aa204ad0d4d947018c4c191e8d05087f P 4ecc8c3409134198b74ab97f3f73384f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:34.286458 15411 catalog_manager.cc:4841] TS 2a77924848fd4c32b7aeb4729b79204d (127.10.131.2:37787): tablet aa204ad0d4d947018c4c191e8d05087f (table test-table [id=5d9fc4e1b3e146638ca2654857a59b28]) successfully deleted
I20260620 18:32:34.294020 15913 ts_tablet_manager.cc:1918] T aa204ad0d4d947018c4c191e8d05087f P 4ecc8c3409134198b74ab97f3f73384f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:32:34.294206 15913 log.cc:1196] T aa204ad0d4d947018c4c191e8d05087f P 4ecc8c3409134198b74ab97f3f73384f: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781980314095189-10764-0/minicluster-data/ts-2/wal/wals/aa204ad0d4d947018c4c191e8d05087f
I20260620 18:32:34.294811 15913 ts_tablet_manager.cc:1939] T aa204ad0d4d947018c4c191e8d05087f P 4ecc8c3409134198b74ab97f3f73384f: Deleting consensus metadata
I20260620 18:32:34.296695 15409 catalog_manager.cc:4841] TS 4ecc8c3409134198b74ab97f3f73384f (127.10.131.3:41161): tablet aa204ad0d4d947018c4c191e8d05087f (table test-table [id=5d9fc4e1b3e146638ca2654857a59b28]) successfully deleted
I20260620 18:32:34.369695 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 15483
W20260620 18:32:34.388564 15408 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.131.1:40753 (error 108)
I20260620 18:32:34.388764 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 15614
I20260620 18:32:34.390785 15887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:34.390321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:40753 (local address 127.0.0.1:37742)
0620 18:32:34.390478 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:34.390605 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 18:32:34.391002 15408 catalog_manager.cc:4584] TS a536c3d7cf4946fbb5277765ab3442c0 (127.10.131.1:40753): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aa204ad0d4d947018c4c191e8d05087f: Network error: Client connection negotiation failed: client connection to 127.10.131.1:40753: connect: Connection refused (error 111)
I20260620 18:32:34.392216 15885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:34.389752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51690 (local address 127.10.131.2:37787)
0620 18:32:34.389953 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:34.389960 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:34.389982 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:32:34.390069 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:34.390074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:34.390177 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:34.390282 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:34.391589 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:34.391596 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:34.391601 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:32:34.391620 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:34.391661 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:32:34.391676 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:34.391799 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:34.391876 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 18:32:34.391933 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:32:34.393074 15736 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:34.444628 15887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:34.444087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:40753 (local address 127.0.0.1:37754)
0620 18:32:34.444347 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:34.444452 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 18:32:34.445823 15887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:34.445405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:40753 (local address 127.0.0.1:37758)
0620 18:32:34.445575 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:34.445666 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 18:32:34.479314 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 15614
I20260620 18:32:34.501235 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 15744
I20260620 18:32:34.504231 15886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:34.502206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59194 (local address 127.10.131.3:41161)
0620 18:32:34.502407 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:34.502412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:34.502435 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:32:34.502507 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:34.502510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:34.502559 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:34.502645 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:34.503545 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:34.503549 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:34.503553 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:32:34.503567 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:34.503597 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:32:34.503608 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:34.503719 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:34.503837 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 18:32:34.503888 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:32:34.505086 15866 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:34.506413 15887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:34.505969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:40753 (local address 127.0.0.1:37774)
0620 18:32:34.506169 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:34.506258 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 18:32:34.507597 15887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:34.507119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:40753 (local address 127.0.0.1:37782)
0620 18:32:34.507272 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:34.507420 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 18:32:34.588286 15887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:34.587754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:40753 (local address 127.0.0.1:37792)
0620 18:32:34.588000 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:34.588116 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 18:32:34.589459 15887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:34.589020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:40753 (local address 127.0.0.1:37796)
0620 18:32:34.589232 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:34.589322 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260620 18:32:34.621984 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 15744
I20260620 18:32:34.640270 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 15393
I20260620 18:32:34.641553 15454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:34.745994 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 15393
2026-06-20T18:32:34Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2635 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 18:32:34.787681 10764 mini_oidc.cc:152] Starting JWKS server
I20260620 18:32:34.790246 10764 webserver.cc:460] Webserver started at https://127.0.0.1:32941/ using document root <none> and password file <none>
I20260620 18:32:34.790571 10764 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 18:32:34.791270 10764 webserver.cc:460] Webserver started at http://127.0.0.1:46363/ using document root <none> and password file <none>
2026-06-20T18:32:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:32:34Z Disabled control of system clock
I20260620 18:32:34.818167 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:38853
--webserver_interface=127.10.131.62
--webserver_port=0
--builtin_ntp_servers=127.10.131.20:33073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:38853
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32941/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980314095189-10764-0/testchainca.cert with env {}
W20260620 18:32:35.136305 15926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:35.136615 15926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:35.136701 15926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:35.143874 15926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:32:35.143975 15926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:35.144012 15926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:35.144061 15926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:32:35.144110 15926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 18:32:35.144163 15926 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 18:32:35.144202 15926 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:32941/jwks.json
I20260620 18:32:35.154060 15926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:33073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:38853
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:38853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32941/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980314095189-10764-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:35.155346 15926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:35.156574 15926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:35.168295 15931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:35.168174 15932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:35.168411 15934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:35.169431 15926 server_base.cc:1034] running on GCE node
I20260620 18:32:35.170225 15926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:35.171592 15926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:35.172937 15926 hybrid_clock.cc:648] HybridClock initialized: now 1781980355172807 us; error 48 us; skew 500 ppm
I20260620 18:32:35.173319 15926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:32:35.175613 15926 webserver.cc:460] Webserver started at http://127.10.131.62:32941/ using document root <none> and password file <none>
I20260620 18:32:35.176318 15926 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:35.176467 15926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:35.176788 15926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:35.179682 15926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "af851f3372ce4eafba5a8a9358da5832"
format_stamp: "Formatted at 2026-06-20 18:32:35 on dist-test-slave-cbn3"
server_key: "018e0862d962dd68929fe22708a7484e"
server_key_iv: "efc7c57b5b2d067ad8e688212fb361eb"
server_key_version: "encryptionkey@0"
I20260620 18:32:35.180562 15926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "af851f3372ce4eafba5a8a9358da5832"
format_stamp: "Formatted at 2026-06-20 18:32:35 on dist-test-slave-cbn3"
server_key: "018e0862d962dd68929fe22708a7484e"
server_key_iv: "efc7c57b5b2d067ad8e688212fb361eb"
server_key_version: "encryptionkey@0"
I20260620 18:32:35.185844 15926 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:32:35.189235 15940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:35.190554 15926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 18:32:35.190737 15926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "af851f3372ce4eafba5a8a9358da5832"
format_stamp: "Formatted at 2026-06-20 18:32:35 on dist-test-slave-cbn3"
server_key: "018e0862d962dd68929fe22708a7484e"
server_key_iv: "efc7c57b5b2d067ad8e688212fb361eb"
server_key_version: "encryptionkey@0"
I20260620 18:32:35.190951 15926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:35.211447 15926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:35.212387 15926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:35.212736 15926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:35.231487 15926 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:38853
I20260620 18:32:35.231582 15991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:38853 every 8 connection(s)
I20260620 18:32:35.232817 15926 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:32:35.236354 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 15926
I20260620 18:32:35.236444 15992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:35.236569 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:32:35.237262 10764 external_mini_cluster.cc:1428] Setting key 2ba42248f348f742b8b5c80d228d6264
I20260620 18:32:35.250711 15992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af851f3372ce4eafba5a8a9358da5832: Bootstrap starting.
I20260620 18:32:35.252871 15995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:35.239119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38266 (local address 127.10.131.62:38853)
0620 18:32:35.240206 (+ 1087us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:35.240219 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:35.240269 (+ 50us) server_negotiation.cc:407] Connection header received
0620 18:32:35.242950 (+ 2681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:35.242985 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:35.243574 (+ 589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:35.244016 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:35.244514 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.246328 (+ 1814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.247631 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.248692 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.248899 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:35.250067 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:35.250077 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:35.250095 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:32:35.250793 (+ 698us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:35.250897 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:32:35.250921 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:35.251065 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:35.251218 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 18:32:35.251530 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":331,"threads_started":1}
I20260620 18:32:35.254398 15992 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af851f3372ce4eafba5a8a9358da5832: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:35.255491 15992 log.cc:824] T 00000000000000000000000000000000 P af851f3372ce4eafba5a8a9358da5832: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:35.256752 10764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:32:35.258438 15992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af851f3372ce4eafba5a8a9358da5832: No bootstrap required, opened a new log
I20260620 18:32:35.266155 15992 raft_consensus.cc:348] T 00000000000000000000000000000000 P af851f3372ce4eafba5a8a9358da5832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af851f3372ce4eafba5a8a9358da5832" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38853 } }
I20260620 18:32:35.266538 15992 raft_consensus.cc:374] T 00000000000000000000000000000000 P af851f3372ce4eafba5a8a9358da5832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:35.266654 15992 raft_consensus.cc:729] T 00000000000000000000000000000000 P af851f3372ce4eafba5a8a9358da5832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af851f3372ce4eafba5a8a9358da5832, State: Initialized, Role: FOLLOWER
I20260620 18:32:35.267246 15992 consensus_queue.cc:260] T 00000000000000000000000000000000 P af851f3372ce4eafba5a8a9358da5832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af851f3372ce4eafba5a8a9358da5832" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38853 } }
I20260620 18:32:35.267477 15992 raft_consensus.cc:388] T 00000000000000000000000000000000 P af851f3372ce4eafba5a8a9358da5832 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:32:35.267621 15992 raft_consensus.cc:482] T 00000000000000000000000000000000 P af851f3372ce4eafba5a8a9358da5832 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:32:35.267787 15992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af851f3372ce4eafba5a8a9358da5832 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:35.270314 15992 raft_consensus.cc:504] T 00000000000000000000000000000000 P af851f3372ce4eafba5a8a9358da5832 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af851f3372ce4eafba5a8a9358da5832" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38853 } }
I20260620 18:32:35.270848 15992 leader_election.cc:302] T 00000000000000000000000000000000 P af851f3372ce4eafba5a8a9358da5832 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af851f3372ce4eafba5a8a9358da5832; no voters:
I20260620 18:32:35.271395 15992 leader_election.cc:288] T 00000000000000000000000000000000 P af851f3372ce4eafba5a8a9358da5832 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:32:35.271549 15997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af851f3372ce4eafba5a8a9358da5832 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:35.272262 15997 raft_consensus.cc:686] T 00000000000000000000000000000000 P af851f3372ce4eafba5a8a9358da5832 [term 1 LEADER]: Becoming Leader. State: Replica: af851f3372ce4eafba5a8a9358da5832, State: Running, Role: LEADER
I20260620 18:32:35.272892 15997 consensus_queue.cc:237] T 00000000000000000000000000000000 P af851f3372ce4eafba5a8a9358da5832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af851f3372ce4eafba5a8a9358da5832" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38853 } }
I20260620 18:32:35.273465 15992 sys_catalog.cc:564] T 00000000000000000000000000000000 P af851f3372ce4eafba5a8a9358da5832 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:32:35.276238 15999 sys_catalog.cc:455] T 00000000000000000000000000000000 P af851f3372ce4eafba5a8a9358da5832 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af851f3372ce4eafba5a8a9358da5832. Latest consensus state: current_term: 1 leader_uuid: "af851f3372ce4eafba5a8a9358da5832" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af851f3372ce4eafba5a8a9358da5832" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38853 } } }
I20260620 18:32:35.276221 15998 sys_catalog.cc:455] T 00000000000000000000000000000000 P af851f3372ce4eafba5a8a9358da5832 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af851f3372ce4eafba5a8a9358da5832" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af851f3372ce4eafba5a8a9358da5832" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38853 } } }
I20260620 18:32:35.276636 15998 sys_catalog.cc:458] T 00000000000000000000000000000000 P af851f3372ce4eafba5a8a9358da5832 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:35.276640 15999 sys_catalog.cc:458] T 00000000000000000000000000000000 P af851f3372ce4eafba5a8a9358da5832 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:35.278241 16003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:32:35.283866 16003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:32:35.293015 16003 catalog_manager.cc:1347] Generated new cluster ID: 10e093997a3347829639a3bf454834ed
I20260620 18:32:35.293190 16003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:32:35.313270 16003 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:32:35.315068 16003 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:32:35.325260 16003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af851f3372ce4eafba5a8a9358da5832: Generated new TSK 0
I20260620 18:32:35.326334 16003 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:32:35.379871 15995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:35.355652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38274 (local address 127.10.131.62:38853)
0620 18:32:35.355847 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:35.355854 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:35.356378 (+ 524us) server_negotiation.cc:407] Connection header received
0620 18:32:35.356726 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:35.356732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:35.356835 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:35.357005 (+ 170us) server_negotiation.cc:226] Negotiated authn=JWT
0620 18:32:35.357588 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.358730 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.360215 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.360947 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.361076 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:35.361608 (+ 532us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 18:32:35.379206 (+ 17598us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32941/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 18:32:35.379573 (+ 367us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38274: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32941/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 18:32:35.380038 15995 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38274: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32941/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 18:32:35.382591 16021 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:32:35.355508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38853 (local address 127.0.0.1:38274)
0620 18:32:35.356249 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:35.356273 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:35.356466 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:35.357086 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:35.357090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:35.357101 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0620 18:32:35.357427 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:35.357436 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.358893 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.358898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:35.360072 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:35.360085 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:35.361139 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:35.361144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:35.361269 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:35.361323 (+ 54us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 18:32:35.380389 (+ 19066us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32941/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 18:32:35.380771 (+ 382us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.131.62:38853: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32941/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":354,"threads_started":1}
E20260620 18:32:35.383113 16022 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 18:32:35.383255 16022 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.10.131.62:38853: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32941/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 18:32:35.384500 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 15926
I20260620 18:32:35.385697 15987 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:35.479988 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 15926
2026-06-20T18:32:35Z chronyd exiting
I20260620 18:32:35.530594 10764 test_util.cc:175] -----------------------------------------------
I20260620 18:32:35.530795 10764 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781980314095189-10764-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (747 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 18:32:35.534848 10764 mini_oidc.cc:152] Starting JWKS server
I20260620 18:32:35.536834 10764 webserver.cc:460] Webserver started at https://127.0.0.1:38987/ using document root <none> and password file <none>
I20260620 18:32:35.537173 10764 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 18:32:35.537886 10764 webserver.cc:460] Webserver started at http://127.0.0.1:43487/ using document root <none> and password file <none>
2026-06-20T18:32:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:32:35Z Disabled control of system clock
I20260620 18:32:35.563930 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:39393
--webserver_interface=127.10.131.62
--webserver_port=0
--builtin_ntp_servers=127.10.131.20:33427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:39393
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38987/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980314095189-10764-0/testchainca.cert with env {}
W20260620 18:32:35.894809 16029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:35.895079 16029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:35.895234 16029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:35.903234 16029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:32:35.903339 16029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:35.903398 16029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:35.903484 16029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:32:35.903573 16029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 18:32:35.903667 16029 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 18:32:35.903738 16029 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38987/jwks.json
I20260620 18:32:35.914022 16029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:33427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:39393
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:39393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38987/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980314095189-10764-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:35.915359 16029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:35.916664 16029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:35.928292 16037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:35.928438 16034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:35.928262 16035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:35.929292 16029 server_base.cc:1034] running on GCE node
I20260620 18:32:35.930186 16029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:35.931581 16029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:35.932829 16029 hybrid_clock.cc:648] HybridClock initialized: now 1781980355932781 us; error 46 us; skew 500 ppm
I20260620 18:32:35.933226 16029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:32:35.935393 16029 webserver.cc:460] Webserver started at http://127.10.131.62:46853/ using document root <none> and password file <none>
I20260620 18:32:35.936066 16029 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:35.936184 16029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:35.936532 16029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:35.939590 16029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "e2fb78285ade4a41b0c916261b87d16a"
format_stamp: "Formatted at 2026-06-20 18:32:35 on dist-test-slave-cbn3"
server_key: "eff47136277768bfaa0f08d70a0f131a"
server_key_iv: "1f0f1947829a1e37597d1fd073854e39"
server_key_version: "encryptionkey@0"
I20260620 18:32:35.940371 16029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "e2fb78285ade4a41b0c916261b87d16a"
format_stamp: "Formatted at 2026-06-20 18:32:35 on dist-test-slave-cbn3"
server_key: "eff47136277768bfaa0f08d70a0f131a"
server_key_iv: "1f0f1947829a1e37597d1fd073854e39"
server_key_version: "encryptionkey@0"
I20260620 18:32:35.945660 16029 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 18:32:35.949355 16043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:35.950856 16029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:32:35.951066 16029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "e2fb78285ade4a41b0c916261b87d16a"
format_stamp: "Formatted at 2026-06-20 18:32:35 on dist-test-slave-cbn3"
server_key: "eff47136277768bfaa0f08d70a0f131a"
server_key_iv: "1f0f1947829a1e37597d1fd073854e39"
server_key_version: "encryptionkey@0"
I20260620 18:32:35.951296 16029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:35.978300 16029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:35.979182 16029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:35.979512 16029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:35.997826 16029 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:39393
I20260620 18:32:35.997844 16094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:39393 every 8 connection(s)
I20260620 18:32:35.999130 16029 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:32:36.002725 16095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:36.003281 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 16029
I20260620 18:32:36.003489 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:32:36.004118 10764 external_mini_cluster.cc:1428] Setting key c5de5b1c0d5d4295802522fd20253930
I20260620 18:32:36.015935 16095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2fb78285ade4a41b0c916261b87d16a: Bootstrap starting.
I20260620 18:32:36.017781 16098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:36.005926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58144 (local address 127.10.131.62:39393)
0620 18:32:36.006925 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:36.006938 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:36.006980 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:32:36.008538 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:36.008559 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:36.009069 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:36.009397 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:36.009998 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:36.011253 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:36.013186 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:36.013922 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:36.014069 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:36.015268 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:36.015274 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:36.015288 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:32:36.015813 (+ 525us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:36.015892 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:32:36.015912 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:36.016013 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:36.016339 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 18:32:36.016682 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":374,"threads_started":1}
I20260620 18:32:36.019759 16095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2fb78285ade4a41b0c916261b87d16a: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:36.021266 16095 log.cc:824] T 00000000000000000000000000000000 P e2fb78285ade4a41b0c916261b87d16a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:36.021946 10764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:32:36.024058 16095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2fb78285ade4a41b0c916261b87d16a: No bootstrap required, opened a new log
I20260620 18:32:36.031113 16095 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2fb78285ade4a41b0c916261b87d16a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2fb78285ade4a41b0c916261b87d16a" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 39393 } }
I20260620 18:32:36.031497 16095 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2fb78285ade4a41b0c916261b87d16a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:36.031613 16095 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2fb78285ade4a41b0c916261b87d16a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2fb78285ade4a41b0c916261b87d16a, State: Initialized, Role: FOLLOWER
I20260620 18:32:36.032220 16095 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2fb78285ade4a41b0c916261b87d16a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2fb78285ade4a41b0c916261b87d16a" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 39393 } }
I20260620 18:32:36.032461 16095 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2fb78285ade4a41b0c916261b87d16a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:32:36.032622 16095 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2fb78285ade4a41b0c916261b87d16a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:32:36.032784 16095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2fb78285ade4a41b0c916261b87d16a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:36.034891 16095 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2fb78285ade4a41b0c916261b87d16a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2fb78285ade4a41b0c916261b87d16a" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 39393 } }
I20260620 18:32:36.035387 16095 leader_election.cc:302] T 00000000000000000000000000000000 P e2fb78285ade4a41b0c916261b87d16a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2fb78285ade4a41b0c916261b87d16a; no voters:
I20260620 18:32:36.036052 16095 leader_election.cc:288] T 00000000000000000000000000000000 P e2fb78285ade4a41b0c916261b87d16a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:32:36.036186 16100 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2fb78285ade4a41b0c916261b87d16a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:36.037201 16100 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2fb78285ade4a41b0c916261b87d16a [term 1 LEADER]: Becoming Leader. State: Replica: e2fb78285ade4a41b0c916261b87d16a, State: Running, Role: LEADER
I20260620 18:32:36.037771 16100 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2fb78285ade4a41b0c916261b87d16a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2fb78285ade4a41b0c916261b87d16a" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 39393 } }
I20260620 18:32:36.038472 16095 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2fb78285ade4a41b0c916261b87d16a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:32:36.041534 16101 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2fb78285ade4a41b0c916261b87d16a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2fb78285ade4a41b0c916261b87d16a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2fb78285ade4a41b0c916261b87d16a" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 39393 } } }
I20260620 18:32:36.041888 16101 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2fb78285ade4a41b0c916261b87d16a [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:36.045091 16102 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2fb78285ade4a41b0c916261b87d16a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2fb78285ade4a41b0c916261b87d16a. Latest consensus state: current_term: 1 leader_uuid: "e2fb78285ade4a41b0c916261b87d16a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2fb78285ade4a41b0c916261b87d16a" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 39393 } } }
I20260620 18:32:36.045629 16102 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2fb78285ade4a41b0c916261b87d16a [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:36.045953 16107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:32:36.050948 16107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:32:36.059388 16107 catalog_manager.cc:1347] Generated new cluster ID: 3e752a24942141ae84acbccdeb3b9186
I20260620 18:32:36.059504 16107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:32:36.083972 16107 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:32:36.085366 16107 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:32:36.095706 16107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2fb78285ade4a41b0c916261b87d16a: Generated new TSK 0
I20260620 18:32:36.096568 16107 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 18:32:36.134219 16125 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 18:32:36.134356 16125 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 18:32:36.135417 16098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:36.115175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58154 (local address 127.10.131.62:39393)
0620 18:32:36.115365 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:36.115371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:36.115777 (+ 406us) server_negotiation.cc:407] Connection header received
0620 18:32:36.115975 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:36.115980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:36.116069 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:36.116233 (+ 164us) server_negotiation.cc:226] Negotiated authn=JWT
0620 18:32:36.116732 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:36.118032 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:36.119470 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:36.120196 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:36.120340 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:36.120871 (+ 531us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 18:32:36.134874 (+ 14003us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38987/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 18:32:36.135185 (+ 311us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58154: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38987/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 18:32:36.135567 16098 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58154: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38987/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 18:32:36.135855 16124 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:32:36.115052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:39393 (local address 127.0.0.1:58154)
0620 18:32:36.115678 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:36.115694 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:36.115814 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:36.116247 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:36.116250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:36.116259 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0620 18:32:36.116525 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:36.116533 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:36.118213 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:36.118217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:36.119321 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:36.119333 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:36.120382 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:36.120387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:36.120529 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:36.120567 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 18:32:36.135134 (+ 14567us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38987/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 18:32:36.135435 (+ 301us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.131.62:39393: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38987/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":357,"threads_started":1}
I20260620 18:32:36.136925 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 16029
I20260620 18:32:36.138303 16090 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:36.231256 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 16029
2026-06-20T18:32:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (751 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 18:32:36.286590 10764 mini_oidc.cc:152] Starting JWKS server
I20260620 18:32:36.288390 10764 webserver.cc:460] Webserver started at https://127.0.0.1:37189/ using document root <none> and password file <none>
I20260620 18:32:36.288575 10764 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 18:32:36.289223 10764 webserver.cc:460] Webserver started at http://127.0.0.1:36079/ using document root <none> and password file <none>
2026-06-20T18:32:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:32:36Z Disabled control of system clock
I20260620 18:32:36.315563 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:35547
--webserver_interface=127.10.131.62
--webserver_port=0
--builtin_ntp_servers=127.10.131.20:37209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:35547
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37189/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 18:32:36.631383 16132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:36.631742 16132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:36.631899 16132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:36.639160 16132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:32:36.639307 16132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:36.639397 16132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:36.639501 16132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:32:36.639585 16132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 18:32:36.639671 16132 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 18:32:36.639753 16132 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37189/jwks.json
I20260620 18:32:36.648757 16132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:37209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:35547
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:35547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37189/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:36.650084 16132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:36.651842 16132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:36.663336 16140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:36.663336 16138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:36.663645 16137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:36.664094 16132 server_base.cc:1034] running on GCE node
I20260620 18:32:36.664958 16132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:36.666291 16132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:36.667526 16132 hybrid_clock.cc:648] HybridClock initialized: now 1781980356667486 us; error 52 us; skew 500 ppm
I20260620 18:32:36.667900 16132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:32:36.670169 16132 webserver.cc:460] Webserver started at http://127.10.131.62:33525/ using document root <none> and password file <none>
I20260620 18:32:36.670887 16132 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:36.671017 16132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:36.671336 16132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:36.674182 16132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "318b702c9aff4d2fbdc74a73131b4e31"
format_stamp: "Formatted at 2026-06-20 18:32:36 on dist-test-slave-cbn3"
server_key: "8521c3f2678dbfd0874f187f83a15bec"
server_key_iv: "90cf6db28aa4dfb5a5b5858f72b58017"
server_key_version: "encryptionkey@0"
I20260620 18:32:36.674981 16132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "318b702c9aff4d2fbdc74a73131b4e31"
format_stamp: "Formatted at 2026-06-20 18:32:36 on dist-test-slave-cbn3"
server_key: "8521c3f2678dbfd0874f187f83a15bec"
server_key_iv: "90cf6db28aa4dfb5a5b5858f72b58017"
server_key_version: "encryptionkey@0"
I20260620 18:32:36.679894 16132 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:32:36.683085 16146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:36.684350 16132 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 18:32:36.684528 16132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "318b702c9aff4d2fbdc74a73131b4e31"
format_stamp: "Formatted at 2026-06-20 18:32:36 on dist-test-slave-cbn3"
server_key: "8521c3f2678dbfd0874f187f83a15bec"
server_key_iv: "90cf6db28aa4dfb5a5b5858f72b58017"
server_key_version: "encryptionkey@0"
I20260620 18:32:36.684753 16132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:36.703924 16132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:36.704728 16132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:36.705046 16132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:36.723264 16132 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:35547
I20260620 18:32:36.723275 16197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:35547 every 8 connection(s)
I20260620 18:32:36.724582 16132 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:32:36.728183 16198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:36.732679 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 16132
I20260620 18:32:36.732908 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:32:36.733547 10764 external_mini_cluster.cc:1428] Setting key af0be9d84da795faad653255a98b71c6
I20260620 18:32:36.738541 16198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 318b702c9aff4d2fbdc74a73131b4e31: Bootstrap starting.
I20260620 18:32:36.742043 16198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 318b702c9aff4d2fbdc74a73131b4e31: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:36.743216 16198 log.cc:824] T 00000000000000000000000000000000 P 318b702c9aff4d2fbdc74a73131b4e31: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:36.746338 16198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 318b702c9aff4d2fbdc74a73131b4e31: No bootstrap required, opened a new log
I20260620 18:32:36.747483 16201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:36.735413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50668 (local address 127.10.131.62:35547)
0620 18:32:36.736473 (+ 1060us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:36.736487 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:36.736530 (+ 43us) server_negotiation.cc:407] Connection header received
0620 18:32:36.738197 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:36.738218 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:36.738641 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:36.739012 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:36.739589 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:36.740923 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:36.742796 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:36.743523 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:36.743679 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:36.745039 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:36.745046 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:36.745059 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:32:36.745575 (+ 516us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:36.745655 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:32:36.745675 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:36.745767 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:36.745993 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 18:32:36.746308 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":384,"threads_started":1}
I20260620 18:32:36.751642 10764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:32:36.755517 16198 raft_consensus.cc:348] T 00000000000000000000000000000000 P 318b702c9aff4d2fbdc74a73131b4e31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "318b702c9aff4d2fbdc74a73131b4e31" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35547 } }
I20260620 18:32:36.755892 16198 raft_consensus.cc:374] T 00000000000000000000000000000000 P 318b702c9aff4d2fbdc74a73131b4e31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:36.756006 16198 raft_consensus.cc:729] T 00000000000000000000000000000000 P 318b702c9aff4d2fbdc74a73131b4e31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 318b702c9aff4d2fbdc74a73131b4e31, State: Initialized, Role: FOLLOWER
I20260620 18:32:36.756564 16198 consensus_queue.cc:260] T 00000000000000000000000000000000 P 318b702c9aff4d2fbdc74a73131b4e31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "318b702c9aff4d2fbdc74a73131b4e31" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35547 } }
I20260620 18:32:36.756764 16198 raft_consensus.cc:388] T 00000000000000000000000000000000 P 318b702c9aff4d2fbdc74a73131b4e31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:32:36.756963 16198 raft_consensus.cc:482] T 00000000000000000000000000000000 P 318b702c9aff4d2fbdc74a73131b4e31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:32:36.757095 16198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 318b702c9aff4d2fbdc74a73131b4e31 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:36.759135 16198 raft_consensus.cc:504] T 00000000000000000000000000000000 P 318b702c9aff4d2fbdc74a73131b4e31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "318b702c9aff4d2fbdc74a73131b4e31" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35547 } }
I20260620 18:32:36.759618 16198 leader_election.cc:302] T 00000000000000000000000000000000 P 318b702c9aff4d2fbdc74a73131b4e31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 318b702c9aff4d2fbdc74a73131b4e31; no voters:
I20260620 18:32:36.760129 16198 leader_election.cc:288] T 00000000000000000000000000000000 P 318b702c9aff4d2fbdc74a73131b4e31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:32:36.760288 16203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 318b702c9aff4d2fbdc74a73131b4e31 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:36.761715 16203 raft_consensus.cc:686] T 00000000000000000000000000000000 P 318b702c9aff4d2fbdc74a73131b4e31 [term 1 LEADER]: Becoming Leader. State: Replica: 318b702c9aff4d2fbdc74a73131b4e31, State: Running, Role: LEADER
I20260620 18:32:36.762274 16198 sys_catalog.cc:564] T 00000000000000000000000000000000 P 318b702c9aff4d2fbdc74a73131b4e31 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:32:36.762228 16203 consensus_queue.cc:237] T 00000000000000000000000000000000 P 318b702c9aff4d2fbdc74a73131b4e31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "318b702c9aff4d2fbdc74a73131b4e31" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35547 } }
I20260620 18:32:36.765064 16204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 318b702c9aff4d2fbdc74a73131b4e31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "318b702c9aff4d2fbdc74a73131b4e31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "318b702c9aff4d2fbdc74a73131b4e31" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35547 } } }
I20260620 18:32:36.765136 16205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 318b702c9aff4d2fbdc74a73131b4e31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 318b702c9aff4d2fbdc74a73131b4e31. Latest consensus state: current_term: 1 leader_uuid: "318b702c9aff4d2fbdc74a73131b4e31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "318b702c9aff4d2fbdc74a73131b4e31" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35547 } } }
I20260620 18:32:36.765498 16204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 318b702c9aff4d2fbdc74a73131b4e31 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:36.765498 16205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 318b702c9aff4d2fbdc74a73131b4e31 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:36.767477 16212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:32:36.771893 16212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:32:36.779930 16212 catalog_manager.cc:1347] Generated new cluster ID: ace0fc82cd614e34a788b6bbc5fb86ac
I20260620 18:32:36.780040 16212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:32:36.798816 16212 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:32:36.800318 16212 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:32:36.807096 16212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 318b702c9aff4d2fbdc74a73131b4e31: Generated new TSK 0
I20260620 18:32:36.808295 16212 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:32:36.830654 16201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:36.822337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50676 (local address 127.10.131.62:35547)
0620 18:32:36.822503 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:36.822508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:36.822891 (+ 383us) server_negotiation.cc:407] Connection header received
0620 18:32:36.823119 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:36.823122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:36.823199 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:36.823344 (+ 145us) server_negotiation.cc:226] Negotiated authn=JWT
0620 18:32:36.823930 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:36.824638 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:36.826630 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:36.827088 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:36.827182 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:36.827769 (+ 587us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 18:32:36.830025 (+ 2256us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37189/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 18:32:36.830326 (+ 301us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50676: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37189/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 18:32:36.830801 16201 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50676: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37189/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 18:32:36.832858 16227 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:32:36.822155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35547 (local address 127.0.0.1:50676)
0620 18:32:36.822778 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:36.822798 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:36.822958 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:36.823395 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:36.823401 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:36.823414 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0620 18:32:36.823757 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:36.823772 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:36.824769 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:36.824775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:36.826471 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:36.826488 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:36.827280 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:36.827287 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:36.827472 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:36.827519 (+ 47us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 18:32:36.830228 (+ 2709us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37189/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 18:32:36.832355 (+ 2127us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.131.62:35547: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37189/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":1822,"thread_start_us":339,"threads_started":1}
I20260620 18:32:36.834276 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 16132
I20260620 18:32:36.835409 16193 generic_service.cc:196] Checking for leaks (request via RPC)
E20260620 18:32:36.846310 16228 webserver.cc:533] Webserver: sslize failed:
E20260620 18:32:36.846455 16228 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 18:32:36.927819 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 16132
2026-06-20T18:32:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (695 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T18:32:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:32:36Z Disabled control of system clock
I20260620 18:32:37.007970 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:35791
--webserver_interface=127.10.131.62
--webserver_port=0
--builtin_ntp_servers=127.10.131.20:41517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:35791
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980314095189-10764-0/testchainca.cert with env {}
W20260620 18:32:37.320416 16233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:37.320698 16233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:37.320784 16233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:37.327898 16233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:32:37.327998 16233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:37.328038 16233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:37.328094 16233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:32:37.328136 16233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 18:32:37.328189 16233 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 18:32:37.328227 16233 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 18:32:37.337972 16233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:41517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:35791
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:35791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980314095189-10764-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:37.339248 16233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:37.340488 16233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:37.352412 16239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:37.352406 16241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:37.352952 16238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:37.353261 16233 server_base.cc:1034] running on GCE node
I20260620 18:32:37.354156 16233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:37.355610 16233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:37.356833 16233 hybrid_clock.cc:648] HybridClock initialized: now 1781980357356793 us; error 34 us; skew 500 ppm
I20260620 18:32:37.357249 16233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:32:37.359506 16233 webserver.cc:460] Webserver started at http://127.10.131.62:37709/ using document root <none> and password file <none>
I20260620 18:32:37.360252 16233 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:37.360379 16233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:37.360703 16233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:37.363734 16233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "fd441f9b08ec452c9dee992eabe82fa9"
format_stamp: "Formatted at 2026-06-20 18:32:37 on dist-test-slave-cbn3"
server_key: "7dcac7bd16c59cac4621c7e49cf1f034"
server_key_iv: "4a61983ffef47ff94b7e9563bf29945b"
server_key_version: "encryptionkey@0"
I20260620 18:32:37.364580 16233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "fd441f9b08ec452c9dee992eabe82fa9"
format_stamp: "Formatted at 2026-06-20 18:32:37 on dist-test-slave-cbn3"
server_key: "7dcac7bd16c59cac4621c7e49cf1f034"
server_key_iv: "4a61983ffef47ff94b7e9563bf29945b"
server_key_version: "encryptionkey@0"
I20260620 18:32:37.369755 16233 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:32:37.373191 16247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:37.374477 16233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 18:32:37.374744 16233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "fd441f9b08ec452c9dee992eabe82fa9"
format_stamp: "Formatted at 2026-06-20 18:32:37 on dist-test-slave-cbn3"
server_key: "7dcac7bd16c59cac4621c7e49cf1f034"
server_key_iv: "4a61983ffef47ff94b7e9563bf29945b"
server_key_version: "encryptionkey@0"
I20260620 18:32:37.375015 16233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:37.419296 16233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:37.420199 16233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:37.420513 16233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:37.438750 16233 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:35791
I20260620 18:32:37.438760 16298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:35791 every 8 connection(s)
I20260620 18:32:37.440138 16233 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:32:37.443719 16299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:37.446835 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 16233
I20260620 18:32:37.447028 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:32:37.447698 10764 external_mini_cluster.cc:1428] Setting key 57e0ed973cefb6866c0bedceb6dbda1e
I20260620 18:32:37.455700 16299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd441f9b08ec452c9dee992eabe82fa9: Bootstrap starting.
I20260620 18:32:37.459216 16299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd441f9b08ec452c9dee992eabe82fa9: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:37.460326 16299 log.cc:824] T 00000000000000000000000000000000 P fd441f9b08ec452c9dee992eabe82fa9: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:37.460947 16302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:37.449460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45474 (local address 127.10.131.62:35791)
0620 18:32:37.450357 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:37.450366 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:37.450405 (+ 39us) server_negotiation.cc:407] Connection header received
0620 18:32:37.451918 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:37.451938 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:37.452375 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:37.452679 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:37.453289 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:37.454565 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:37.456311 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:37.457063 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:37.457222 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:37.458433 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:37.458439 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:37.458451 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:32:37.458955 (+ 504us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:37.459025 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:32:37.459044 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:37.459137 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:37.459483 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 18:32:37.459776 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":334,"threads_started":1}
I20260620 18:32:37.463436 16299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd441f9b08ec452c9dee992eabe82fa9: No bootstrap required, opened a new log
I20260620 18:32:37.464684 10764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:32:37.471935 16299 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd441f9b08ec452c9dee992eabe82fa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd441f9b08ec452c9dee992eabe82fa9" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35791 } }
I20260620 18:32:37.472252 16299 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd441f9b08ec452c9dee992eabe82fa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:37.472316 16299 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd441f9b08ec452c9dee992eabe82fa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd441f9b08ec452c9dee992eabe82fa9, State: Initialized, Role: FOLLOWER
I20260620 18:32:37.472877 16299 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd441f9b08ec452c9dee992eabe82fa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd441f9b08ec452c9dee992eabe82fa9" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35791 } }
I20260620 18:32:37.473081 16299 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd441f9b08ec452c9dee992eabe82fa9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:32:37.473178 16299 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd441f9b08ec452c9dee992eabe82fa9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:32:37.473289 16299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd441f9b08ec452c9dee992eabe82fa9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:37.475297 16299 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd441f9b08ec452c9dee992eabe82fa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd441f9b08ec452c9dee992eabe82fa9" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35791 } }
I20260620 18:32:37.475939 16299 leader_election.cc:302] T 00000000000000000000000000000000 P fd441f9b08ec452c9dee992eabe82fa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd441f9b08ec452c9dee992eabe82fa9; no voters:
I20260620 18:32:37.476737 16299 leader_election.cc:288] T 00000000000000000000000000000000 P fd441f9b08ec452c9dee992eabe82fa9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:32:37.476886 16304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd441f9b08ec452c9dee992eabe82fa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:37.477556 16304 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd441f9b08ec452c9dee992eabe82fa9 [term 1 LEADER]: Becoming Leader. State: Replica: fd441f9b08ec452c9dee992eabe82fa9, State: Running, Role: LEADER
I20260620 18:32:37.478094 16304 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd441f9b08ec452c9dee992eabe82fa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd441f9b08ec452c9dee992eabe82fa9" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35791 } }
I20260620 18:32:37.479053 16299 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd441f9b08ec452c9dee992eabe82fa9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:32:37.481206 16306 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd441f9b08ec452c9dee992eabe82fa9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd441f9b08ec452c9dee992eabe82fa9. Latest consensus state: current_term: 1 leader_uuid: "fd441f9b08ec452c9dee992eabe82fa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd441f9b08ec452c9dee992eabe82fa9" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35791 } } }
I20260620 18:32:37.481566 16306 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd441f9b08ec452c9dee992eabe82fa9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:37.481374 16305 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd441f9b08ec452c9dee992eabe82fa9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd441f9b08ec452c9dee992eabe82fa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd441f9b08ec452c9dee992eabe82fa9" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35791 } } }
I20260620 18:32:37.481765 16305 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd441f9b08ec452c9dee992eabe82fa9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:37.489218 16311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:32:37.494150 16311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:32:37.502665 16311 catalog_manager.cc:1347] Generated new cluster ID: 4ef961e208f34e6a982cbb90b9e4b61e
I20260620 18:32:37.502784 16311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:32:37.511889 16311 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:32:37.513638 16311 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:32:37.523286 16311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd441f9b08ec452c9dee992eabe82fa9: Generated new TSK 0
I20260620 18:32:37.524263 16311 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:32:37.602983 16302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:37.539351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45486 (local address 127.10.131.62:35791)
0620 18:32:37.539542 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:37.539549 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:37.540015 (+ 466us) server_negotiation.cc:407] Connection header received
0620 18:32:37.540210 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:37.540216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:37.540303 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:37.540502 (+ 199us) server_negotiation.cc:226] Negotiated authn=JWT
0620 18:32:37.541012 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:37.542162 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:37.543575 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:37.544238 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:37.544389 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:37.544822 (+ 433us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 18:32:37.602222 (+ 57400us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 18:32:37.602650 (+ 428us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45486: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 18:32:37.603130 16302 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45486: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 18:32:37.603175 16328 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:32:37.539195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35791 (local address 127.0.0.1:45486)
0620 18:32:37.539909 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:37.539926 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:37.540051 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:37.540517 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:37.540520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:37.540530 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0620 18:32:37.540771 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:37.540790 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:37.542325 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:37.542329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:37.543437 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:37.543448 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:37.544410 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:37.544415 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:37.544528 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:37.544561 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 18:32:37.602557 (+ 57996us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 18:32:37.602789 (+ 232us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.131.62:35791: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":423,"threads_started":1}
I20260620 18:32:37.604593 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 16233
I20260620 18:32:37.605819 16294 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:37.716768 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 16233
2026-06-20T18:32:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (765 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (398 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1477 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:32:39 dist-test-slave-cbn3 krb5kdc[16344](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:32:39 dist-test-slave-cbn3 krb5kdc[16344](info): set up 2 sockets
Jun 20 18:32:39 dist-test-slave-cbn3 krb5kdc[16344](info): commencing operation
krb5kdc: starting...
W20260620 18:32:40.681643 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.004s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:32:40 dist-test-slave-cbn3 krb5kdc[16344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:32:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:32:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.131.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
I20260620 18:32:40.882014 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:38405
--webserver_interface=127.10.131.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:38405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:41.209709 16357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:41.210027 16357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:41.210117 16357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:41.217621 16357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:32:41.217743 16357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:41.217792 16357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:41.217855 16357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:32:41.217900 16357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:32:41.227845 16357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:38405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:38405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:41.229197 16357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:41.230506 16357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:41.241916 16362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:41.242470 16363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:41.242492 16365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:41.242941 16357 server_base.cc:1034] running on GCE node
I20260620 18:32:41.243660 16357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:41.245064 16357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:41.246264 16357 hybrid_clock.cc:648] HybridClock initialized: now 1781980361246225 us; error 32 us; skew 500 ppm
I20260620 18:32:41.246603 16357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:41 dist-test-slave-cbn3 krb5kdc[16344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980361, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:41.249578 16357 init.cc:377] Logged in from keytab as kudu/127.10.131.62@KRBTEST.COM (short username kudu)
I20260620 18:32:41.251016 16357 webserver.cc:460] Webserver started at http://127.10.131.62:41379/ using document root <none> and password file <none>
I20260620 18:32:41.251646 16357 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:41.251742 16357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:41.252000 16357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:41.254920 16357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "9ec594d7e8e34271b80f685faace1649"
format_stamp: "Formatted at 2026-06-20 18:32:41 on dist-test-slave-cbn3"
server_key: "a8e72d430da0d424a8cde557d682e538"
server_key_iv: "27b58d41344a6066ce0acf2bf14288a3"
server_key_version: "encryptionkey@0"
I20260620 18:32:41.255646 16357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "9ec594d7e8e34271b80f685faace1649"
format_stamp: "Formatted at 2026-06-20 18:32:41 on dist-test-slave-cbn3"
server_key: "a8e72d430da0d424a8cde557d682e538"
server_key_iv: "27b58d41344a6066ce0acf2bf14288a3"
server_key_version: "encryptionkey@0"
I20260620 18:32:41.260660 16357 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260620 18:32:41.264117 16372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:41.265528 16357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:32:41.265658 16357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "9ec594d7e8e34271b80f685faace1649"
format_stamp: "Formatted at 2026-06-20 18:32:41 on dist-test-slave-cbn3"
server_key: "a8e72d430da0d424a8cde557d682e538"
server_key_iv: "27b58d41344a6066ce0acf2bf14288a3"
server_key_version: "encryptionkey@0"
I20260620 18:32:41.265821 16357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:41.286556 16357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:41.290648 16357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:41.290882 16357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:41.309108 16357 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:38405
I20260620 18:32:41.309157 16423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:38405 every 8 connection(s)
I20260620 18:32:41.310468 16357 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:32:41.311009 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 16357
I20260620 18:32:41.311241 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:32:41.312088 10764 external_mini_cluster.cc:1428] Setting key 82cd0769278afe0e82e7cf7dfca8cf12
I20260620 18:32:41.314792 16424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:41.325389 16424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ec594d7e8e34271b80f685faace1649: Bootstrap starting.
Jun 20 18:32:41 dist-test-slave-cbn3 krb5kdc[16344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:32:41.328577 16424 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ec594d7e8e34271b80f685faace1649: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:41.329685 16424 log.cc:824] T 00000000000000000000000000000000 P 9ec594d7e8e34271b80f685faace1649: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:41.332816 16424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ec594d7e8e34271b80f685faace1649: No bootstrap required, opened a new log
I20260620 18:32:41.334012 16427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:41.314122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56976 (local address 127.10.131.62:38405)
0620 18:32:41.314963 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:41.314976 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:41.315022 (+ 46us) server_negotiation.cc:407] Connection header received
0620 18:32:41.316820 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:41.316890 (+ 70us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:41.317324 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:41.317742 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:41.319206 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:41.320546 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:41.321928 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:41.322671 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:41.323003 (+ 332us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:41.326900 (+ 3897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:41.326907 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:41.326921 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:41.326962 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:41.330190 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:41.330948 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:41.330961 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:41.330967 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:41.331120 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:41.331506 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:41.331509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:41.331511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:41.332005 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:41.332222 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:41.332639 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 18:32:41.333049 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":338,"threads_started":1}
I20260620 18:32:41.341296 16424 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ec594d7e8e34271b80f685faace1649 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec594d7e8e34271b80f685faace1649" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38405 } }
I20260620 18:32:41.341710 16424 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ec594d7e8e34271b80f685faace1649 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:41.341806 16424 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ec594d7e8e34271b80f685faace1649 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ec594d7e8e34271b80f685faace1649, State: Initialized, Role: FOLLOWER
I20260620 18:32:41.342348 16424 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ec594d7e8e34271b80f685faace1649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec594d7e8e34271b80f685faace1649" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38405 } }
I20260620 18:32:41.342592 16424 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ec594d7e8e34271b80f685faace1649 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:32:41.342730 16424 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ec594d7e8e34271b80f685faace1649 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:32:41.342922 16424 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ec594d7e8e34271b80f685faace1649 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:41.345155 16424 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ec594d7e8e34271b80f685faace1649 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec594d7e8e34271b80f685faace1649" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38405 } }
I20260620 18:32:41.345599 16424 leader_election.cc:302] T 00000000000000000000000000000000 P 9ec594d7e8e34271b80f685faace1649 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ec594d7e8e34271b80f685faace1649; no voters:
I20260620 18:32:41.346079 16424 leader_election.cc:288] T 00000000000000000000000000000000 P 9ec594d7e8e34271b80f685faace1649 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:32:41.346222 16429 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ec594d7e8e34271b80f685faace1649 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:41.346879 16429 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ec594d7e8e34271b80f685faace1649 [term 1 LEADER]: Becoming Leader. State: Replica: 9ec594d7e8e34271b80f685faace1649, State: Running, Role: LEADER
I20260620 18:32:41.347476 16429 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ec594d7e8e34271b80f685faace1649 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec594d7e8e34271b80f685faace1649" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38405 } }
I20260620 18:32:41.348124 16424 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ec594d7e8e34271b80f685faace1649 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:32:41.350270 16430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ec594d7e8e34271b80f685faace1649 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ec594d7e8e34271b80f685faace1649" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec594d7e8e34271b80f685faace1649" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38405 } } }
I20260620 18:32:41.350710 16430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ec594d7e8e34271b80f685faace1649 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:41.350297 16431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ec594d7e8e34271b80f685faace1649 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ec594d7e8e34271b80f685faace1649. Latest consensus state: current_term: 1 leader_uuid: "9ec594d7e8e34271b80f685faace1649" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec594d7e8e34271b80f685faace1649" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38405 } } }
I20260620 18:32:41.351846 16431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ec594d7e8e34271b80f685faace1649 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:41.353075 16436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:32:41.358336 16436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:32:41.366629 16436 catalog_manager.cc:1347] Generated new cluster ID: b013accac8ac4b73949d8eebd828f9b8
I20260620 18:32:41.366737 16436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:32:41.386569 16436 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:32:41.388441 16436 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:32:41.397389 16436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ec594d7e8e34271b80f685faace1649: Generated new TSK 0
I20260620 18:32:41.398301 16436 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.131.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab.
Entry for principal kudu/127.10.131.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab.
I20260620 18:32:41.460492 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.1:0
--local_ip_for_outbound_sockets=127.10.131.1
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:38405
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:41.788262 16450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:41.788575 16450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:41.788677 16450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:41.796254 16450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:41.796384 16450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:41.796504 16450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.1
I20260620 18:32:41.806473 16450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:38405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.1
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:41.807713 16450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:41.809141 16450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:41.822199 16458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:41.822259 16456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:41.822285 16455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:41.823325 16450 server_base.cc:1034] running on GCE node
I20260620 18:32:41.823848 16450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:41.824679 16450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:41.825891 16450 hybrid_clock.cc:648] HybridClock initialized: now 1781980361825820 us; error 48 us; skew 500 ppm
I20260620 18:32:41.826184 16450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:41 dist-test-slave-cbn3 krb5kdc[16344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980361, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:41.829342 16450 init.cc:377] Logged in from keytab as kudu/127.10.131.1@KRBTEST.COM (short username kudu)
I20260620 18:32:41.830701 16450 webserver.cc:460] Webserver started at http://127.10.131.1:39929/ using document root <none> and password file <none>
I20260620 18:32:41.831470 16450 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:41.831599 16450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:41.831913 16450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:41.834894 16450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-0/data/instance:
uuid: "2b9e00d4a87d434fadb222bbcefeb146"
format_stamp: "Formatted at 2026-06-20 18:32:41 on dist-test-slave-cbn3"
server_key: "5ed18f04d864e0bf3f47f5da67c79a99"
server_key_iv: "76183bb2dd8c9836392a60baa01e2241"
server_key_version: "encryptionkey@0"
I20260620 18:32:41.835726 16450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance:
uuid: "2b9e00d4a87d434fadb222bbcefeb146"
format_stamp: "Formatted at 2026-06-20 18:32:41 on dist-test-slave-cbn3"
server_key: "5ed18f04d864e0bf3f47f5da67c79a99"
server_key_iv: "76183bb2dd8c9836392a60baa01e2241"
server_key_version: "encryptionkey@0"
I20260620 18:32:41.840695 16450 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 18:32:41.844026 16465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:41.845288 16450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:32:41.845458 16450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-0/wal
uuid: "2b9e00d4a87d434fadb222bbcefeb146"
format_stamp: "Formatted at 2026-06-20 18:32:41 on dist-test-slave-cbn3"
server_key: "5ed18f04d864e0bf3f47f5da67c79a99"
server_key_iv: "76183bb2dd8c9836392a60baa01e2241"
server_key_version: "encryptionkey@0"
I20260620 18:32:41.845649 16450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:41.865166 16450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:41.868815 16450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:41.869217 16450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:41.870124 16450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:41.871454 16450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:41.871532 16450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:41.871614 16450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:41.871656 16450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:41.905419 16450 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.1:43001
I20260620 18:32:41.905532 16577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.1:43001 every 8 connection(s)
I20260620 18:32:41.906733 16450 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
I20260620 18:32:41.911075 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 16450
I20260620 18:32:41.911267 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance
I20260620 18:32:41.911896 10764 external_mini_cluster.cc:1428] Setting key 74fba52ef24eca95156ddff04dedb0b3
Jun 20 18:32:41 dist-test-slave-cbn3 krb5kdc[16344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980361, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:32:41.926909 16580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:41.909231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:34937 (local address 127.10.131.62:38405)
0620 18:32:41.909655 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:41.909660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:41.910584 (+ 924us) server_negotiation.cc:407] Connection header received
0620 18:32:41.912607 (+ 2023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:41.912613 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:41.912703 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:41.912819 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:41.914369 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:41.915088 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:41.916697 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:41.917159 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:41.917300 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:41.921291 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:41.921296 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:41.921300 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:41.921333 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:41.923836 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:41.924556 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:41.924563 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:41.924566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:41.924686 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:41.925158 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:41.925164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:41.925168 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:41.925451 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:41.925597 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:41.926291 (+ 694us) server_negotiation.cc:299] Negotiation successful
0620 18:32:41.926534 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":228,"threads_started":1}
I20260620 18:32:41.927897 16581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:41.909713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38405 (local address 127.10.131.1:34937)
0620 18:32:41.910416 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:41.910469 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:41.912297 (+ 1828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:41.913089 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:41.913098 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:41.913617 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:41.914180 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:41.914198 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:41.915259 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:41.915263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:41.916548 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:41.916560 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:41.917399 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:41.917408 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:41.917655 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:41.918347 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:41.918374 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:41.921064 (+ 2690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:41.924009 (+ 2945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:41.924017 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:41.924031 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:41.924438 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:41.924791 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:41.924796 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:41.924800 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:41.925047 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:41.925584 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:41.925593 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:41.925924 (+ 331us) client_negotiation.cc:769] Sending connection context
0620 18:32:41.926189 (+ 265us) client_negotiation.cc:240] Negotiation successful
0620 18:32:41.926723 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":289,"threads_started":1}
I20260620 18:32:41.929548 16578 heartbeater.cc:347] Connected to a master server at 127.10.131.62:38405
I20260620 18:32:41.929896 16578 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.10.131.2@KRBTEST.COM; defaulting to no policy
I20260620 18:32:41.930853 16578 heartbeater.cc:511] Master 127.10.131.62:38405 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.2@KRBTEST.COM" created.
I20260620 18:32:41.933075 16389 ts_manager.cc:194] Registered new tserver with Master: 2b9e00d4a87d434fadb222bbcefeb146 (127.10.131.1:43001)
I20260620 18:32:41.934475 16389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.1@KRBTEST.COM'} at 127.10.131.1:34937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab.
Entry for principal kudu/127.10.131.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab.
I20260620 18:32:41.956336 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.2:0
--local_ip_for_outbound_sockets=127.10.131.2
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:38405
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:42.276247 16584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:42.276530 16584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:42.276685 16584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:42.284196 16584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:42.284327 16584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:42.284507 16584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.2
I20260620 18:32:42.294165 16584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:38405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.2
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:42.295413 16584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:42.296765 16584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:42.309933 16592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:42.310959 16584 server_base.cc:1034] running on GCE node
W20260620 18:32:42.313419 16589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:42.313548 16590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:42.314262 16584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:42.315259 16584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:42.316473 16584 hybrid_clock.cc:648] HybridClock initialized: now 1781980362316430 us; error 38 us; skew 500 ppm
I20260620 18:32:42.316779 16584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:42 dist-test-slave-cbn3 krb5kdc[16344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980362, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:42.319845 16584 init.cc:377] Logged in from keytab as kudu/127.10.131.2@KRBTEST.COM (short username kudu)
I20260620 18:32:42.321404 16584 webserver.cc:460] Webserver started at http://127.10.131.2:34913/ using document root <none> and password file <none>
I20260620 18:32:42.322062 16584 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:42.322184 16584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:42.322481 16584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:42.325459 16584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-1/data/instance:
uuid: "1ae973efc7ec417485654995df6c5e7f"
format_stamp: "Formatted at 2026-06-20 18:32:42 on dist-test-slave-cbn3"
server_key: "a122fc891905f11c96658ccb3a5be9df"
server_key_iv: "f01add63e9cc86bf5d5ec62816c0b746"
server_key_version: "encryptionkey@0"
I20260620 18:32:42.326208 16584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance:
uuid: "1ae973efc7ec417485654995df6c5e7f"
format_stamp: "Formatted at 2026-06-20 18:32:42 on dist-test-slave-cbn3"
server_key: "a122fc891905f11c96658ccb3a5be9df"
server_key_iv: "f01add63e9cc86bf5d5ec62816c0b746"
server_key_version: "encryptionkey@0"
I20260620 18:32:42.331498 16584 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 18:32:42.334897 16599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:42.336264 16584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:32:42.336443 16584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-1/wal
uuid: "1ae973efc7ec417485654995df6c5e7f"
format_stamp: "Formatted at 2026-06-20 18:32:42 on dist-test-slave-cbn3"
server_key: "a122fc891905f11c96658ccb3a5be9df"
server_key_iv: "f01add63e9cc86bf5d5ec62816c0b746"
server_key_version: "encryptionkey@0"
I20260620 18:32:42.336642 16584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:42.350191 16584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:42.354389 16584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:42.354674 16584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:42.355584 16584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:42.356825 16584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:42.356962 16584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:42.357120 16584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:42.357184 16584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:42.389618 16584 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.2:34241
I20260620 18:32:42.389703 16711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.2:34241 every 8 connection(s)
I20260620 18:32:42.390844 16584 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
I20260620 18:32:42.395831 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 16584
I20260620 18:32:42.396036 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance
I20260620 18:32:42.396574 10764 external_mini_cluster.cc:1428] Setting key 8b08d6a3332fdb36bc4fa6e11071c3f5
Jun 20 18:32:42 dist-test-slave-cbn3 krb5kdc[16344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980362, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:32:42.414076 16580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:42.393397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:44471 (local address 127.10.131.62:38405)
0620 18:32:42.393580 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:42.393585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:42.394636 (+ 1051us) server_negotiation.cc:407] Connection header received
0620 18:32:42.396824 (+ 2188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:42.396828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:42.396944 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:42.397032 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:42.398676 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:42.399345 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:42.401823 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:42.402209 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:42.402303 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:42.406976 (+ 4673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:42.406985 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:42.406993 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:42.407055 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:42.410742 (+ 3687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:42.411411 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:42.411419 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:42.411425 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:42.411583 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:42.412089 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:42.412096 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:42.412102 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:42.412469 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:42.412621 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:42.413354 (+ 733us) server_negotiation.cc:299] Negotiation successful
0620 18:32:42.413607 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:32:42.414722 16714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:42.393765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38405 (local address 127.10.131.2:44471)
0620 18:32:42.394435 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:42.394507 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:42.396524 (+ 2017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:42.397317 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:42.397331 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:42.397938 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:42.398495 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:42.398519 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:42.399502 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:42.399511 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:42.401689 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:42.401707 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:42.402423 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:42.402432 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:42.402758 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:42.403476 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:42.403510 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:42.406693 (+ 3183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:42.410878 (+ 4185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:42.410885 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:42.410898 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:42.411294 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:42.411782 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:42.411786 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:42.411788 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:42.411975 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:42.412609 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:42.412614 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:42.412989 (+ 375us) client_negotiation.cc:769] Sending connection context
0620 18:32:42.413229 (+ 240us) client_negotiation.cc:240] Negotiation successful
0620 18:32:42.413614 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":262,"threads_started":1}
I20260620 18:32:42.416164 16712 heartbeater.cc:347] Connected to a master server at 127.10.131.62:38405
I20260620 18:32:42.416463 16712 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:42.417234 16712 heartbeater.cc:511] Master 127.10.131.62:38405 requested a full tablet report, sending...
I20260620 18:32:42.418609 16389 ts_manager.cc:194] Registered new tserver with Master: 1ae973efc7ec417485654995df6c5e7f (127.10.131.2:34241)
WARNING: no policy specified for kudu/127.10.131.3@KRBTEST.COM; defaulting to no policy
I20260620 18:32:42.419427 16389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.2@KRBTEST.COM'} at 127.10.131.2:44471
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab.
Entry for principal kudu/127.10.131.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab.
I20260620 18:32:42.444516 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.3:0
--local_ip_for_outbound_sockets=127.10.131.3
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:38405
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:42.762320 16717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:42.762635 16717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:42.762804 16717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:42.770430 16717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:42.770587 16717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:42.770756 16717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.3
I20260620 18:32:42.780522 16717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:38405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.3
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:42.781852 16717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:42.783152 16717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:42.796271 16725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:42.796353 16722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:42.796314 16723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:42.797343 16717 server_base.cc:1034] running on GCE node
I20260620 18:32:42.797986 16717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:42.798924 16717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:42.800166 16717 hybrid_clock.cc:648] HybridClock initialized: now 1781980362800107 us; error 44 us; skew 500 ppm
I20260620 18:32:42.800515 16717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:42 dist-test-slave-cbn3 krb5kdc[16344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980362, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:42.803853 16717 init.cc:377] Logged in from keytab as kudu/127.10.131.3@KRBTEST.COM (short username kudu)
I20260620 18:32:42.805282 16717 webserver.cc:460] Webserver started at http://127.10.131.3:33079/ using document root <none> and password file <none>
I20260620 18:32:42.805933 16717 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:42.806071 16717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:42.806448 16717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:42.809345 16717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-2/data/instance:
uuid: "c3a0f1f73415469896e19698685e489d"
format_stamp: "Formatted at 2026-06-20 18:32:42 on dist-test-slave-cbn3"
server_key: "1ead16887933099cbe9ac69c8a8164e9"
server_key_iv: "182f1a9acdd2c7563879aa4f153b5511"
server_key_version: "encryptionkey@0"
I20260620 18:32:42.810106 16717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance:
uuid: "c3a0f1f73415469896e19698685e489d"
format_stamp: "Formatted at 2026-06-20 18:32:42 on dist-test-slave-cbn3"
server_key: "1ead16887933099cbe9ac69c8a8164e9"
server_key_iv: "182f1a9acdd2c7563879aa4f153b5511"
server_key_version: "encryptionkey@0"
I20260620 18:32:42.815099 16717 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:32:42.818522 16732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:42.819779 16717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 18:32:42.819943 16717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-2/wal
uuid: "c3a0f1f73415469896e19698685e489d"
format_stamp: "Formatted at 2026-06-20 18:32:42 on dist-test-slave-cbn3"
server_key: "1ead16887933099cbe9ac69c8a8164e9"
server_key_iv: "182f1a9acdd2c7563879aa4f153b5511"
server_key_version: "encryptionkey@0"
I20260620 18:32:42.820178 16717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:42.857147 16717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:42.861588 16717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:42.861878 16717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:42.862815 16717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:42.864168 16717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:42.864277 16717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:42.864414 16717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:42.864482 16717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:42.898051 16717 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.3:40019
I20260620 18:32:42.898137 16844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.3:40019 every 8 connection(s)
I20260620 18:32:42.899335 16717 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
I20260620 18:32:42.904119 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 16717
I20260620 18:32:42.904275 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance
I20260620 18:32:42.904874 10764 external_mini_cluster.cc:1428] Setting key 34873ca2531923b694b0ecb6a0ab4ec3
Jun 20 18:32:42 dist-test-slave-cbn3 krb5kdc[16344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980362, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:32:42.919170 16580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:42.901791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:51017 (local address 127.10.131.62:38405)
0620 18:32:42.901969 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:42.901973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:42.903083 (+ 1110us) server_negotiation.cc:407] Connection header received
0620 18:32:42.905171 (+ 2088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:42.905177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:42.905293 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:42.905404 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:42.907905 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:42.909030 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:42.910476 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:42.910911 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:42.911008 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:42.914089 (+ 3081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:42.914094 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:42.914097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:42.914127 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:42.916432 (+ 2305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:42.917119 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:42.917123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:42.917125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:42.917247 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:42.917597 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:42.917600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:42.917602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:42.917843 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:42.917946 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:42.918626 (+ 680us) server_negotiation.cc:299] Negotiation successful
0620 18:32:42.918824 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:32:42.920132 16847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:42.902273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38405 (local address 127.10.131.3:51017)
0620 18:32:42.902924 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:42.902960 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:42.904821 (+ 1861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:42.905628 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:42.905637 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:42.906338 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:42.907025 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:42.907052 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:42.909162 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:42.909166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:42.910342 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:42.910353 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:42.911126 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:42.911131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:42.911307 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:42.911904 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:42.911924 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:42.913916 (+ 1992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:42.916547 (+ 2631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:42.916553 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:42.916584 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:42.917024 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:42.917332 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:42.917335 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:42.917338 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:42.917489 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:42.917934 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:42.917939 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:42.918217 (+ 278us) client_negotiation.cc:769] Sending connection context
0620 18:32:42.918497 (+ 280us) client_negotiation.cc:240] Negotiation successful
0620 18:32:42.918948 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":273,"threads_started":1}
I20260620 18:32:42.921363 16845 heartbeater.cc:347] Connected to a master server at 127.10.131.62:38405
I20260620 18:32:42.921648 16845 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:42.922335 16845 heartbeater.cc:511] Master 127.10.131.62:38405 requested a full tablet report, sending...
I20260620 18:32:42.923651 16389 ts_manager.cc:194] Registered new tserver with Master: c3a0f1f73415469896e19698685e489d (127.10.131.3:40019)
I20260620 18:32:42.924427 16389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.3@KRBTEST.COM'} at 127.10.131.3:51017
I20260620 18:32:42.932294 10764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:32:42.937453 16578 heartbeater.cc:503] Master 127.10.131.62:38405 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/krb5kdc/test-admin.keytab.
Jun 20 18:32:42 dist-test-slave-cbn3 krb5kdc[16344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:42.955937 10764 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 18:32:42.960963 16852 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:32:42.958487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37510 (local address 127.10.131.1:43001)
0620 18:32:42.959127 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:42.959142 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:42.959166 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:32:42.959223 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:42.959229 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:42.959430 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:42.959614 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:42.960475 (+ 861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37510: BlockingRecv error: recv got EOF from 127.0.0.1:37510 (error 108)
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":296,"threads_started":1}
W20260620 18:32:42.961002 10764 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/krb5kdc/krb5cc' of type FILE
Jun 20 18:32:42 dist-test-slave-cbn3 krb5kdc[16344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:42.962486 10764 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 18:32:42 dist-test-slave-cbn3 krb5kdc[16344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:42.963752 10764 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 18:32:42 dist-test-slave-cbn3 krb5kdc[16344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.1@KRBTEST.COM
I20260620 18:32:42.978315 16852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:42.965164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37512 (local address 127.10.131.1:43001)
0620 18:32:42.965363 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:42.965368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:42.965608 (+ 240us) server_negotiation.cc:407] Connection header received
0620 18:32:42.965789 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:42.965794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:42.965857 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:42.965952 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:42.966882 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:42.967808 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:42.969052 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:42.969657 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:42.969785 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:42.973069 (+ 3284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:42.973087 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:42.973102 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:42.973131 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:42.975693 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:42.976329 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:42.976337 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:42.976341 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:42.976460 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:42.976802 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:42.976809 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:42.976812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:42.977174 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:42.977467 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:42.977794 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 18:32:42.978038 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 18:32:42.983453 16852 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:32:42.981936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37518 (local address 127.10.131.1:43001)
0620 18:32:42.982143 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:42.982148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:42.982335 (+ 187us) server_negotiation.cc:407] Connection header received
0620 18:32:42.982454 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:42.982457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:42.982510 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:42.982583 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:42.983238 (+ 655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37518: BlockingRecv error: recv got EOF from 127.0.0.1:37518 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 18:32:42.983631 10764 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/krb5kdc/krb5cc' of type FILE
Jun 20 18:32:42 dist-test-slave-cbn3 krb5kdc[16344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:42.985008 10764 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 18:32:42 dist-test-slave-cbn3 krb5kdc[16344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:42.986230 10764 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 18:32:42 dist-test-slave-cbn3 krb5kdc[16344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.1@KRBTEST.COM
I20260620 18:32:42.999298 16852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:42.987536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37524 (local address 127.10.131.1:43001)
0620 18:32:42.987701 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:42.987705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:42.987773 (+ 68us) server_negotiation.cc:407] Connection header received
0620 18:32:42.987924 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:42.987928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:42.987982 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:42.988072 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:42.988951 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:42.989641 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:42.991096 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:42.991508 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:42.991600 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:42.994703 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:42.994707 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:42.994711 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:42.994738 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:42.997265 (+ 2527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:42.997804 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:42.997807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:42.997809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:42.997918 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:42.998215 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:42.998218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:42.998220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:42.998447 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:42.998623 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:42.998848 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 18:32:42.999016 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 18:32:43.003021 16852 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:32:43.001661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37526 (local address 127.10.131.1:43001)
0620 18:32:43.001849 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:43.001854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:43.001996 (+ 142us) server_negotiation.cc:407] Connection header received
0620 18:32:43.002118 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:43.002121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:43.002183 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:43.002252 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:43.002813 (+ 561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37526: BlockingRecv error: recv got EOF from 127.0.0.1:37526 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 18:32:43.003170 10764 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781980314095189-10764-0/krb5kdc/krb5cc' of type FILE
Jun 20 18:32:42 dist-test-slave-cbn3 krb5kdc[16344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:43.004508 10764 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 18:32:43 dist-test-slave-cbn3 krb5kdc[16344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:43.005729 10764 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 18:32:43 dist-test-slave-cbn3 krb5kdc[16344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.1@KRBTEST.COM
I20260620 18:32:43.018476 16852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:43.006876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37532 (local address 127.10.131.1:43001)
0620 18:32:43.007068 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:43.007073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:43.007265 (+ 192us) server_negotiation.cc:407] Connection header received
0620 18:32:43.007366 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:43.007369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:43.007416 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:43.007486 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:43.008385 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:43.009115 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:43.010291 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:43.010667 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:43.010752 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:43.013913 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:43.013917 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:43.013920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:43.013947 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:43.016372 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:43.016939 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:43.016943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:43.016945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:43.017088 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:43.017374 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:43.017377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:43.017379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:43.017624 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:43.017831 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:43.017995 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 18:32:43.018184 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:32:43.019918 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 16450
I20260620 18:32:43.030819 16852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:43.020625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37546 (local address 127.10.131.1:43001)
0620 18:32:43.020824 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:43.020832 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:43.021298 (+ 466us) server_negotiation.cc:407] Connection header received
0620 18:32:43.021410 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:43.021413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:43.021463 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:43.021533 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:43.022311 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:43.022924 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:43.024135 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:43.024522 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:43.024622 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:43.026479 (+ 1857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:43.026483 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:43.026486 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:43.026511 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:43.028781 (+ 2270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:43.029393 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:43.029396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:43.029399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:43.029510 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:43.029825 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:43.029828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:43.029830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:43.030046 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:43.030237 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:43.030440 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 18:32:43.030579 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 18:32:43.031383 16573 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:43.112993 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 16450
I20260620 18:32:43.129045 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 16584
Jun 20 18:32:43 dist-test-slave-cbn3 krb5kdc[16344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.2@KRBTEST.COM
I20260620 18:32:43.143695 16865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:43.129906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56660 (local address 127.10.131.2:34241)
0620 18:32:43.130498 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:43.130505 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:43.130530 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:32:43.130591 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:43.130599 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:43.130808 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:43.130937 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:43.131795 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:43.132805 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:43.134018 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:43.134551 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:43.134645 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:43.138185 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:43.138196 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:43.138210 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:43.138241 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:43.141271 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:43.141962 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:43.141969 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:43.141971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:43.142090 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:43.142417 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:43.142420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:43.142422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:43.142741 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:43.142961 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:43.143220 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 18:32:43.143417 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":295,"threads_started":1}
I20260620 18:32:43.144934 16707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:43.224790 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 16584
I20260620 18:32:43.240741 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 16717
Jun 20 18:32:43 dist-test-slave-cbn3 krb5kdc[16344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.3@KRBTEST.COM
I20260620 18:32:43.255679 16867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:43.241595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49770 (local address 127.10.131.3:40019)
0620 18:32:43.242205 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:43.242212 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:43.242257 (+ 45us) server_negotiation.cc:407] Connection header received
0620 18:32:43.242317 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:43.242323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:43.242523 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:43.242652 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:43.243651 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:43.244669 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:43.246031 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:43.246548 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:43.246639 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:43.250512 (+ 3873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:43.250522 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:43.250534 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:43.250564 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:43.253421 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:43.254050 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:43.254057 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:43.254059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:43.254167 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:43.254466 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:43.254469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:43.254471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:43.254791 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:43.255013 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:43.255193 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 18:32:43.255410 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":303,"threads_started":1}
I20260620 18:32:43.256930 16840 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:43.336833 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 16717
I20260620 18:32:43.352082 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 16357
I20260620 18:32:43.353179 16419 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:43.452812 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 16357
2026-06-20T18:32:43Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3868 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:32:43 dist-test-slave-cbn3 krb5kdc[16872](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:32:43 dist-test-slave-cbn3 krb5kdc[16872](info): set up 2 sockets
Jun 20 18:32:43 dist-test-slave-cbn3 krb5kdc[16872](info): commencing operation
krb5kdc: starting...
W20260620 18:32:44.537691 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:32:44 dist-test-slave-cbn3 krb5kdc[16872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:32:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:32:44Z Disabled control of system clock
WARNING: no policy specified for oryx/127.10.131.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.131.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.62.keytab.
Entry for principal oryx/127.10.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.62.keytab.
I20260620 18:32:44.734392 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:44649
--webserver_interface=127.10.131.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.62.keytab
--principal=oryx/127.10.131.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:37493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:44649
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:45.048173 16885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:45.048444 16885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:45.048532 16885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:45.055652 16885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:32:45.055759 16885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:45.055806 16885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:45.055871 16885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:32:45.055913 16885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:32:45.065485 16885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:37493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:44649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.62.keytab
--principal=oryx/127.10.131.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:44649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:45.066699 16885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:45.067948 16885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:45.079383 16890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:45.079439 16891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:45.079442 16893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:45.080051 16885 server_base.cc:1034] running on GCE node
I20260620 18:32:45.080922 16885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:45.082285 16885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:45.083503 16885 hybrid_clock.cc:648] HybridClock initialized: now 1781980365083451 us; error 42 us; skew 500 ppm
I20260620 18:32:45.083838 16885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:45 dist-test-slave-cbn3 krb5kdc[16872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980365, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:45.086833 16885 init.cc:377] Logged in from keytab as oryx/127.10.131.62@KRBTEST.COM (short username oryx)
I20260620 18:32:45.088229 16885 webserver.cc:460] Webserver started at http://127.10.131.62:35987/ using document root <none> and password file <none>
I20260620 18:32:45.088946 16885 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:45.089082 16885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:45.089370 16885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:45.092182 16885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "2d7e501282a949a1be211a121201e250"
format_stamp: "Formatted at 2026-06-20 18:32:45 on dist-test-slave-cbn3"
server_key: "a650855f436d737b8d9f48f37565f83c"
server_key_iv: "11b67d322a66df10dda149c5418150f8"
server_key_version: "encryptionkey@0"
I20260620 18:32:45.093022 16885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "2d7e501282a949a1be211a121201e250"
format_stamp: "Formatted at 2026-06-20 18:32:45 on dist-test-slave-cbn3"
server_key: "a650855f436d737b8d9f48f37565f83c"
server_key_iv: "11b67d322a66df10dda149c5418150f8"
server_key_version: "encryptionkey@0"
I20260620 18:32:45.098172 16885 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 18:32:45.101477 16900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:45.102787 16885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:32:45.103073 16885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "2d7e501282a949a1be211a121201e250"
format_stamp: "Formatted at 2026-06-20 18:32:45 on dist-test-slave-cbn3"
server_key: "a650855f436d737b8d9f48f37565f83c"
server_key_iv: "11b67d322a66df10dda149c5418150f8"
server_key_version: "encryptionkey@0"
I20260620 18:32:45.103291 16885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:45.118325 16885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:45.122164 16885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:45.122430 16885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:45.140237 16885 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:44649
I20260620 18:32:45.140267 16951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:44649 every 8 connection(s)
I20260620 18:32:45.141619 16885 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:32:45.143298 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 16885
I20260620 18:32:45.143512 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:32:45.144372 10764 external_mini_cluster.cc:1428] Setting key 8c7aaf7569475951a7b562d95f4fd216
I20260620 18:32:45.146086 16952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:32:45 dist-test-slave-cbn3 krb5kdc[16872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM
I20260620 18:32:45.159616 16952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250: Bootstrap starting.
I20260620 18:32:45.162864 16952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:45.163880 16952 log.cc:824] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:45.166324 16955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:45.146412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57880 (local address 127.10.131.62:44649)
0620 18:32:45.147179 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:45.147190 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:45.147228 (+ 38us) server_negotiation.cc:407] Connection header received
0620 18:32:45.148718 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:45.148738 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:45.149179 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:45.149497 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:45.151247 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:45.152537 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:45.154245 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:45.154884 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:45.155033 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:45.159539 (+ 4506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:45.159547 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:45.159561 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:45.159593 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:45.162528 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:45.163211 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:45.163218 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:45.163224 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:45.163383 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:45.163739 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:45.163744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:45.163747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:45.164279 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:45.164608 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:45.165036 (+ 428us) server_negotiation.cc:299] Negotiation successful
0620 18:32:45.165398 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":366,"threads_started":1}
I20260620 18:32:45.166762 16952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250: No bootstrap required, opened a new log
I20260620 18:32:45.174310 16952 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } }
I20260620 18:32:45.174686 16952 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:45.174788 16952 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d7e501282a949a1be211a121201e250, State: Initialized, Role: FOLLOWER
I20260620 18:32:45.175388 16952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } }
I20260620 18:32:45.175634 16952 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:32:45.175770 16952 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:32:45.175941 16952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:45.178423 16952 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } }
I20260620 18:32:45.178885 16952 leader_election.cc:302] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d7e501282a949a1be211a121201e250; no voters:
I20260620 18:32:45.179440 16952 leader_election.cc:288] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:32:45.179555 16957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:45.180222 16957 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [term 1 LEADER]: Becoming Leader. State: Replica: 2d7e501282a949a1be211a121201e250, State: Running, Role: LEADER
I20260620 18:32:45.180773 16957 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } }
I20260620 18:32:45.181563 16952 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:32:45.183265 16958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d7e501282a949a1be211a121201e250" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } }
I20260620 18:32:45.183617 16958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:45.184190 16959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d7e501282a949a1be211a121201e250. Latest consensus state: current_term: 1 leader_uuid: "2d7e501282a949a1be211a121201e250" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } }
I20260620 18:32:45.184582 16959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:45.184583 16963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:32:45.189945 16963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:32:45.198992 16963 catalog_manager.cc:1347] Generated new cluster ID: e971777715374237a5b255fc27e8cc4a
I20260620 18:32:45.199090 16963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:32:45.209362 16963 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:32:45.210582 16963 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:32:45.219892 16963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250: Generated new TSK 0
I20260620 18:32:45.220678 16963 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.10.131.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.131.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.131.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.1.keytab.
Entry for principal oryx/127.10.131.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.1.keytab.
I20260620 18:32:45.278756 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.1:0
--local_ip_for_outbound_sockets=127.10.131.1
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:44649
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.1.keytab
--principal=oryx/127.10.131.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:37493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:45.635834 16978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:45.636132 16978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:45.636276 16978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:45.643671 16978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:45.643786 16978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:45.643976 16978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.1
I20260620 18:32:45.653761 16978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:37493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.1.keytab
--principal=oryx/127.10.131.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:44649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.1
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:45.655066 16978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:45.656375 16978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:45.669725 16984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:45.669783 16983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:45.669807 16986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:45.671058 16978 server_base.cc:1034] running on GCE node
I20260620 18:32:45.671734 16978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:45.672668 16978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:45.673898 16978 hybrid_clock.cc:648] HybridClock initialized: now 1781980365673843 us; error 74 us; skew 500 ppm
I20260620 18:32:45.674218 16978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:45 dist-test-slave-cbn3 krb5kdc[16872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980365, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:45.677567 16978 init.cc:377] Logged in from keytab as oryx/127.10.131.1@KRBTEST.COM (short username oryx)
I20260620 18:32:45.679041 16978 webserver.cc:460] Webserver started at http://127.10.131.1:44583/ using document root <none> and password file <none>
I20260620 18:32:45.679658 16978 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:45.679759 16978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:45.680003 16978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:45.683095 16978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-0/data/instance:
uuid: "2694a3ddafbc484e882015fb46f9bbfc"
format_stamp: "Formatted at 2026-06-20 18:32:45 on dist-test-slave-cbn3"
server_key: "fcf2359cb5bca6a0d00eee86e1c57bc3"
server_key_iv: "de625206061749554c6164b817804e0b"
server_key_version: "encryptionkey@0"
I20260620 18:32:45.683796 16978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance:
uuid: "2694a3ddafbc484e882015fb46f9bbfc"
format_stamp: "Formatted at 2026-06-20 18:32:45 on dist-test-slave-cbn3"
server_key: "fcf2359cb5bca6a0d00eee86e1c57bc3"
server_key_iv: "de625206061749554c6164b817804e0b"
server_key_version: "encryptionkey@0"
I20260620 18:32:45.688696 16978 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 18:32:45.692179 16993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:45.693545 16978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:32:45.693672 16978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-0/wal
uuid: "2694a3ddafbc484e882015fb46f9bbfc"
format_stamp: "Formatted at 2026-06-20 18:32:45 on dist-test-slave-cbn3"
server_key: "fcf2359cb5bca6a0d00eee86e1c57bc3"
server_key_iv: "de625206061749554c6164b817804e0b"
server_key_version: "encryptionkey@0"
I20260620 18:32:45.693831 16978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:45.713517 16978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:45.717777 16978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:45.718029 16978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:45.718931 16978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:45.720264 16978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:45.720338 16978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:45.720495 16978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:45.720553 16978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:45.753630 16978 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.1:45237
I20260620 18:32:45.753729 17105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.1:45237 every 8 connection(s)
I20260620 18:32:45.754937 16978 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
I20260620 18:32:45.762574 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 16978
I20260620 18:32:45.762835 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance
I20260620 18:32:45.763471 10764 external_mini_cluster.cc:1428] Setting key d6d81fb69f968c8afa24c4accbef51e9
Jun 20 18:32:45 dist-test-slave-cbn3 krb5kdc[16872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980365, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM
I20260620 18:32:45.777132 17108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:45.757570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:38659 (local address 127.10.131.62:44649)
0620 18:32:45.758062 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:45.758067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:45.759054 (+ 987us) server_negotiation.cc:407] Connection header received
0620 18:32:45.760750 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:45.760754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:45.760833 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:45.760984 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:45.762694 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:45.763444 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:45.765557 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:45.765938 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:45.766040 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:45.770507 (+ 4467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:45.770512 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:45.770516 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:45.770550 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:45.773665 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:45.774660 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:45.774663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:45.774666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:45.774781 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:45.775311 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:45.775314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:45.775316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:45.775576 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:45.775695 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:45.776627 (+ 932us) server_negotiation.cc:299] Negotiation successful
0620 18:32:45.776801 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":217,"threads_started":1}
I20260620 18:32:45.778740 17109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:45.758159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.10.131.1:38659)
0620 18:32:45.758854 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:45.758922 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:45.760497 (+ 1575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:45.761190 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:45.761198 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:45.761733 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:45.762478 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:45.762505 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:45.763614 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:45.763620 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:45.765372 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:45.765385 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:45.766174 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:45.766183 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:45.766521 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:45.767309 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:45.767350 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:45.770276 (+ 2926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:45.773824 (+ 3548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:45.773841 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:45.773862 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:45.774542 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:45.774888 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:45.774896 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:45.774903 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:45.775203 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:45.775709 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:45.775721 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:45.776232 (+ 511us) client_negotiation.cc:769] Sending connection context
0620 18:32:45.776549 (+ 317us) client_negotiation.cc:240] Negotiation successful
0620 18:32:45.777185 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":259,"threads_started":1}
I20260620 18:32:45.780467 17106 heartbeater.cc:347] Connected to a master server at 127.10.131.62:44649
I20260620 18:32:45.780802 17106 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:45.781641 17106 heartbeater.cc:511] Master 127.10.131.62:44649 requested a full tablet report, sending...
I20260620 18:32:45.784188 16917 ts_manager.cc:194] Registered new tserver with Master: 2694a3ddafbc484e882015fb46f9bbfc (127.10.131.1:45237)
I20260620 18:32:45.786487 16917 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.131.1@KRBTEST.COM'} at 127.10.131.1:38659
WARNING: no policy specified for oryx/127.10.131.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.131.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.131.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.2.keytab.
Entry for principal oryx/127.10.131.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.2.keytab.
I20260620 18:32:45.815680 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.2:0
--local_ip_for_outbound_sockets=127.10.131.2
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:44649
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.2.keytab
--principal=oryx/127.10.131.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:37493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:46.131551 17112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:46.131820 17112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:46.131911 17112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:46.139071 17112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:46.139179 17112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:46.139292 17112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.2
I20260620 18:32:46.154525 17112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:37493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.2.keytab
--principal=oryx/127.10.131.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:44649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.2
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:46.156009 17112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:46.157539 17112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:46.170177 17117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:46.170202 17120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:46.170260 17118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:46.171434 17112 server_base.cc:1034] running on GCE node
I20260620 18:32:46.172004 17112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:46.172986 17112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:46.174176 17112 hybrid_clock.cc:648] HybridClock initialized: now 1781980366174132 us; error 42 us; skew 500 ppm
I20260620 18:32:46.174441 17112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:46 dist-test-slave-cbn3 krb5kdc[16872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980366, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:46.181360 17112 init.cc:377] Logged in from keytab as oryx/127.10.131.2@KRBTEST.COM (short username oryx)
I20260620 18:32:46.182904 17112 webserver.cc:460] Webserver started at http://127.10.131.2:46551/ using document root <none> and password file <none>
I20260620 18:32:46.183538 17112 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:46.183640 17112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:46.183873 17112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:46.186904 17112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-1/data/instance:
uuid: "f6e2a79583ed46ad9d079b7a72b70b30"
format_stamp: "Formatted at 2026-06-20 18:32:46 on dist-test-slave-cbn3"
server_key: "c491c797896a90bc1897f15c163df968"
server_key_iv: "b7be3747a434117b10205a539c1c2143"
server_key_version: "encryptionkey@0"
I20260620 18:32:46.187645 17112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance:
uuid: "f6e2a79583ed46ad9d079b7a72b70b30"
format_stamp: "Formatted at 2026-06-20 18:32:46 on dist-test-slave-cbn3"
server_key: "c491c797896a90bc1897f15c163df968"
server_key_iv: "b7be3747a434117b10205a539c1c2143"
server_key_version: "encryptionkey@0"
I20260620 18:32:46.193091 17112 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 18:32:46.196677 17127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:46.198235 17112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:32:46.198392 17112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-1/wal
uuid: "f6e2a79583ed46ad9d079b7a72b70b30"
format_stamp: "Formatted at 2026-06-20 18:32:46 on dist-test-slave-cbn3"
server_key: "c491c797896a90bc1897f15c163df968"
server_key_iv: "b7be3747a434117b10205a539c1c2143"
server_key_version: "encryptionkey@0"
I20260620 18:32:46.198585 17112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:46.216907 17112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:46.221181 17112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:46.221459 17112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:46.222398 17112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:46.223733 17112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:46.223829 17112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:46.223969 17112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:46.224035 17112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:46.257195 17112 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.2:35827
I20260620 18:32:46.257279 17239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.2:35827 every 8 connection(s)
I20260620 18:32:46.258468 17112 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
I20260620 18:32:46.265784 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 17112
I20260620 18:32:46.265969 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance
I20260620 18:32:46.266433 10764 external_mini_cluster.cc:1428] Setting key eebbedbda340ba9632bddb763c17d342
Jun 20 18:32:46 dist-test-slave-cbn3 krb5kdc[16872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980366, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.2@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM
I20260620 18:32:46.278599 17108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:46.261208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:59315 (local address 127.10.131.62:44649)
0620 18:32:46.261380 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:46.261385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:46.262406 (+ 1021us) server_negotiation.cc:407] Connection header received
0620 18:32:46.264146 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:46.264150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:46.264217 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:46.264499 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:46.266064 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:46.266850 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:46.268391 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:46.268790 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:46.268980 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:46.272337 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:46.272341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:46.272345 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:46.272375 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:46.275046 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:46.276049 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:46.276061 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:46.276068 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:46.276238 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:46.276772 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:46.276776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:46.276778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:46.277132 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:46.277255 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:46.278183 (+ 928us) server_negotiation.cc:299] Negotiation successful
0620 18:32:46.278362 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:32:46.280143 17242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:46.261563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.10.131.2:59315)
0620 18:32:46.262248 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:46.262288 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:46.263874 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:46.264735 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:46.264743 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:46.265322 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:46.265882 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:46.265908 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:46.266999 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:46.267006 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:46.268248 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:46.268260 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:46.269056 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:46.269065 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:46.269292 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:46.269993 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:46.270020 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:46.272117 (+ 2097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:46.275231 (+ 3114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:46.275243 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:46.275264 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:46.275907 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:46.276393 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:46.276401 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:46.276408 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:46.276655 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:46.277259 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:46.277270 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:46.277771 (+ 501us) client_negotiation.cc:769] Sending connection context
0620 18:32:46.278095 (+ 324us) client_negotiation.cc:240] Negotiation successful
0620 18:32:46.278680 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":271,"threads_started":1}
I20260620 18:32:46.281570 17240 heartbeater.cc:347] Connected to a master server at 127.10.131.62:44649
I20260620 18:32:46.281904 17240 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:46.282651 17240 heartbeater.cc:511] Master 127.10.131.62:44649 requested a full tablet report, sending...
I20260620 18:32:46.284192 16917 ts_manager.cc:194] Registered new tserver with Master: f6e2a79583ed46ad9d079b7a72b70b30 (127.10.131.2:35827)
I20260620 18:32:46.285080 16917 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.131.2@KRBTEST.COM'} at 127.10.131.2:59315
WARNING: no policy specified for oryx/127.10.131.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.131.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.131.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.3.keytab.
Entry for principal oryx/127.10.131.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.3.keytab.
I20260620 18:32:46.314205 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.3:0
--local_ip_for_outbound_sockets=127.10.131.3
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:44649
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.3.keytab
--principal=oryx/127.10.131.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:37493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:46.642452 17245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:46.642719 17245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:46.642812 17245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:46.649972 17245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:46.650090 17245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:46.650202 17245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.3
I20260620 18:32:46.659616 17245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:37493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.3.keytab
--principal=oryx/127.10.131.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:44649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.3
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:46.660830 17245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:46.662056 17245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:46.674615 17250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:46.674571 17253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:46.674646 17251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:46.675566 17245 server_base.cc:1034] running on GCE node
I20260620 18:32:46.676198 17245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:46.677153 17245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:46.678369 17245 hybrid_clock.cc:648] HybridClock initialized: now 1781980366678325 us; error 52 us; skew 500 ppm
I20260620 18:32:46.678668 17245 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:46 dist-test-slave-cbn3 krb5kdc[16872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980366, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:46.681773 17245 init.cc:377] Logged in from keytab as oryx/127.10.131.3@KRBTEST.COM (short username oryx)
I20260620 18:32:46.683320 17245 webserver.cc:460] Webserver started at http://127.10.131.3:44981/ using document root <none> and password file <none>
I20260620 18:32:46.683964 17245 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:46.684121 17245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:46.684412 17245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:46.687230 17245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-2/data/instance:
uuid: "4e122b7d3c6f4e4b934949448ccd2fa5"
format_stamp: "Formatted at 2026-06-20 18:32:46 on dist-test-slave-cbn3"
server_key: "13cf5633ce4997965f8f4d21a811bab6"
server_key_iv: "cce1f7f207eeaeecdb1e697ee63c77d4"
server_key_version: "encryptionkey@0"
I20260620 18:32:46.687986 17245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance:
uuid: "4e122b7d3c6f4e4b934949448ccd2fa5"
format_stamp: "Formatted at 2026-06-20 18:32:46 on dist-test-slave-cbn3"
server_key: "13cf5633ce4997965f8f4d21a811bab6"
server_key_iv: "cce1f7f207eeaeecdb1e697ee63c77d4"
server_key_version: "encryptionkey@0"
I20260620 18:32:46.693358 17245 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:32:46.696507 17260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:46.697880 17245 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:32:46.698062 17245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-2/wal
uuid: "4e122b7d3c6f4e4b934949448ccd2fa5"
format_stamp: "Formatted at 2026-06-20 18:32:46 on dist-test-slave-cbn3"
server_key: "13cf5633ce4997965f8f4d21a811bab6"
server_key_iv: "cce1f7f207eeaeecdb1e697ee63c77d4"
server_key_version: "encryptionkey@0"
I20260620 18:32:46.698258 17245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:46.712051 17245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:46.716401 17245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:46.716653 17245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:46.717664 17245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:46.718956 17245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:46.719028 17245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:46.719115 17245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:46.719164 17245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:46.752132 17245 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.3:34005
I20260620 18:32:46.752177 17372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.3:34005 every 8 connection(s)
I20260620 18:32:46.753487 17245 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
I20260620 18:32:46.754766 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 17245
I20260620 18:32:46.754953 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance
I20260620 18:32:46.755518 10764 external_mini_cluster.cc:1428] Setting key 39e57c19e463bdbc75a5670b823b909c
Jun 20 18:32:46 dist-test-slave-cbn3 krb5kdc[16872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980366, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.3@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM
I20260620 18:32:46.775511 17108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:46.758001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:44543 (local address 127.10.131.62:44649)
0620 18:32:46.758174 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:46.758180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:46.759833 (+ 1653us) server_negotiation.cc:407] Connection header received
0620 18:32:46.761514 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:46.761518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:46.761594 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:46.761676 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:46.763017 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:46.763763 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:46.765272 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:46.765816 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:46.765920 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:46.769312 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:46.769317 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:46.769320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:46.769354 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:46.772444 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:46.773232 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:46.773238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:46.773241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:46.773371 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:46.773793 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:46.773798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:46.773801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:46.774084 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:46.774232 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:46.774934 (+ 702us) server_negotiation.cc:299] Negotiation successful
0620 18:32:46.775163 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:32:46.776494 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:46.758823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.10.131.3:44543)
0620 18:32:46.759690 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:46.759726 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:46.761284 (+ 1558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:46.761861 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:46.761868 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:46.762346 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:46.762875 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:46.762891 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:46.763884 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:46.763888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:46.765147 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:46.765157 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:46.766016 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:46.766021 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:46.766214 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:46.766859 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:46.766882 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:46.769098 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:46.772594 (+ 3496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:46.772603 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:46.772620 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:46.773114 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:46.773480 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:46.773485 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:46.773489 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:46.773677 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:46.774208 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:46.774216 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:46.774559 (+ 343us) client_negotiation.cc:769] Sending connection context
0620 18:32:46.774826 (+ 267us) client_negotiation.cc:240] Negotiation successful
0620 18:32:46.775358 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":353,"threads_started":1}
I20260620 18:32:46.777860 17373 heartbeater.cc:347] Connected to a master server at 127.10.131.62:44649
I20260620 18:32:46.778172 17373 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:46.778898 17373 heartbeater.cc:511] Master 127.10.131.62:44649 requested a full tablet report, sending...
I20260620 18:32:46.780207 16917 ts_manager.cc:194] Registered new tserver with Master: 4e122b7d3c6f4e4b934949448ccd2fa5 (127.10.131.3:34005)
I20260620 18:32:46.781183 16917 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.131.3@KRBTEST.COM'} at 127.10.131.3:44543
I20260620 18:32:46.783653 10764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:32:46.789711 17106 heartbeater.cc:503] Master 127.10.131.62:44649 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.10.131.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.131.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.131.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.61.keytab.
Entry for principal oryx/127.10.131.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.61.keytab.
I20260620 18:32:46.832237 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.61:38067
--webserver_interface=127.10.131.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.61.keytab
--principal=oryx/127.10.131.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:37493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:44649,127.10.131.61:38067
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:47.157006 17378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:47.157284 17378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:47.157371 17378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:47.164400 17378 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:32:47.164507 17378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:47.164553 17378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:47.164606 17378 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:32:47.164650 17378 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:32:47.174474 17378 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:37493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:44649,127.10.131.61:38067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.61.keytab
--principal=oryx/127.10.131.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.61:38067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.131.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
W20260620 18:32:47.174798 17378 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.10.131.62:44649,127.10.131.61:38067'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 18:32:47.175786 17378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:47.177052 17378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:47.189056 17383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:47.188974 17384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:47.189010 17386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:47.189646 17378 server_base.cc:1034] running on GCE node
I20260620 18:32:47.190519 17378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:47.192111 17378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:47.193361 17378 hybrid_clock.cc:648] HybridClock initialized: now 1781980367193286 us; error 54 us; skew 500 ppm
I20260620 18:32:47.193711 17378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:47 dist-test-slave-cbn3 krb5kdc[16872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980367, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:47.197021 17378 init.cc:377] Logged in from keytab as oryx/127.10.131.61@KRBTEST.COM (short username oryx)
I20260620 18:32:47.198568 17378 webserver.cc:460] Webserver started at http://127.10.131.61:44991/ using document root <none> and password file <none>
I20260620 18:32:47.199270 17378 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:47.199395 17378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:47.199713 17378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:47.202976 17378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-1/data/instance:
uuid: "888f00220e5f44c19e0e7f0bff6a1ca9"
format_stamp: "Formatted at 2026-06-20 18:32:47 on dist-test-slave-cbn3"
server_key: "063f761768fc0aae2948d566a36ecb91"
server_key_iv: "6ac329e5143944778f6c0dbdbb3610b1"
server_key_version: "encryptionkey@0"
I20260620 18:32:47.203748 17378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-1/wal/instance:
uuid: "888f00220e5f44c19e0e7f0bff6a1ca9"
format_stamp: "Formatted at 2026-06-20 18:32:47 on dist-test-slave-cbn3"
server_key: "063f761768fc0aae2948d566a36ecb91"
server_key_iv: "6ac329e5143944778f6c0dbdbb3610b1"
server_key_version: "encryptionkey@0"
I20260620 18:32:47.209316 17378 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 18:32:47.212780 17393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:47.214287 17378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:32:47.214473 17378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-1/wal
uuid: "888f00220e5f44c19e0e7f0bff6a1ca9"
format_stamp: "Formatted at 2026-06-20 18:32:47 on dist-test-slave-cbn3"
server_key: "063f761768fc0aae2948d566a36ecb91"
server_key_iv: "6ac329e5143944778f6c0dbdbb3610b1"
server_key_version: "encryptionkey@0"
I20260620 18:32:47.214702 17378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:47.228077 17378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:47.232337 17378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:47.232613 17378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:47.250669 17378 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.61:38067
I20260620 18:32:47.250711 17444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.61:38067 every 8 connection(s)
I20260620 18:32:47.251992 17378 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-1/data/info.pb
I20260620 18:32:47.252904 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 17378
I20260620 18:32:47.256258 17445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:47.264622 17445 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:32:47 dist-test-slave-cbn3 krb5kdc[16872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.131.61@KRBTEST.COM
Jun 20 18:32:47 dist-test-slave-cbn3 krb5kdc[16872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980367, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.61@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM
I20260620 18:32:47.280035 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.255253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41910 (local address 127.10.131.61:38067)
0620 18:32:47.256007 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:47.256020 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:47.256061 (+ 41us) server_negotiation.cc:407] Connection header received
0620 18:32:47.258868 (+ 2807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:47.258897 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:47.259440 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:47.259913 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:47.261823 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.263581 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.265363 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.266096 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.266272 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:47.271321 (+ 5049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:47.271336 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:47.271350 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:47.271397 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:47.275575 (+ 4178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:47.276414 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:47.276424 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:47.276431 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:47.276595 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:47.277225 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:47.277229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:47.277231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:47.277742 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:47.278097 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:47.278607 (+ 510us) server_negotiation.cc:299] Negotiation successful
0620 18:32:47.278967 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":338,"threads_started":1}
I20260620 18:32:47.283516 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.267362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.0.0.1:57894)
0620 18:32:47.268180 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:47.268228 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:47.268418 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:47.268793 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:47.268803 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:47.269640 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:47.269962 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:47.269971 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.270990 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.270997 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:47.272702 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:47.272716 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.273674 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.273682 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:47.273879 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:47.274573 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:47.274600 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:47.277456 (+ 2856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:47.280796 (+ 3340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:47.280808 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:47.280815 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:47.281365 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:47.281733 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:47.281737 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:47.281740 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:47.281898 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:47.282660 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:47.282683 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:47.282879 (+ 196us) client_negotiation.cc:769] Sending connection context
0620 18:32:47.283033 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 18:32:47.283255 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":363,"threads_started":1}
I20260620 18:32:47.283571 17108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.267410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57894 (local address 127.10.131.62:44649)
0620 18:32:47.267660 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:47.267670 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:47.268355 (+ 685us) server_negotiation.cc:407] Connection header received
0620 18:32:47.268535 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:47.268540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:47.268612 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:47.268693 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:47.270102 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.270867 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.272895 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.273472 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.273568 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:47.277634 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:47.277640 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:47.277643 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:47.277676 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:47.280674 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:47.281486 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:47.281490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:47.281492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:47.281620 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:47.282042 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:47.282047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:47.282049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:47.282513 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:47.282708 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:47.283150 (+ 442us) server_negotiation.cc:299] Negotiation successful
0620 18:32:47.283352 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:32:47.285421 17445 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:32:47.287669 17240 heartbeater.cc:503] Master 127.10.131.62:44649 was elected leader, sending a full tablet report...
Jun 20 18:32:47 dist-test-slave-cbn3 krb5kdc[16872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980367, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.61@KRBTEST.COM for oryx/127.10.131.61@KRBTEST.COM
I20260620 18:32:47.298520 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.286304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:38067 (local address 127.0.0.1:41916)
0620 18:32:47.286496 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:47.286515 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:47.286648 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:47.286931 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:47.286935 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:47.287278 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:47.287402 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:47.287410 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.288501 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.288506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:47.289694 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:47.289705 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.290480 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.290485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:47.290640 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:47.291291 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:47.291310 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:47.293468 (+ 2158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:47.296078 (+ 2610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:47.296082 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:47.296085 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:47.296508 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:47.296936 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:47.296940 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:47.296942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:47.297075 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:47.297549 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:47.297553 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:47.297700 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 18:32:47.297885 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 18:32:47.298153 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 18:32:47.298681 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.286305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41916 (local address 127.10.131.61:38067)
0620 18:32:47.286496 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:47.286503 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:47.286589 (+ 86us) server_negotiation.cc:407] Connection header received
0620 18:32:47.286771 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:47.286780 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:47.286829 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:47.286918 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:47.287528 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.288331 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.289821 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.290329 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.290422 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:47.293618 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:47.293623 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:47.293626 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:47.293652 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:47.295945 (+ 2293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:47.296612 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:47.296616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:47.296620 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:47.296762 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:47.297177 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:47.297180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:47.297182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:47.297426 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:47.297583 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:47.297947 (+ 364us) server_negotiation.cc:299] Negotiation successful
0620 18:32:47.298316 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":149,"server-negotiator.queue_time_us":65}
I20260620 18:32:47.303331 17445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9: Bootstrap starting.
I20260620 18:32:47.306314 17445 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:47.307324 17445 log.cc:824] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:47.309895 17445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9: No bootstrap required, opened a new log
I20260620 18:32:47.316602 17445 raft_consensus.cc:348] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } }
I20260620 18:32:47.317030 17445 raft_consensus.cc:374] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:47.317137 17445 raft_consensus.cc:729] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 888f00220e5f44c19e0e7f0bff6a1ca9, State: Initialized, Role: FOLLOWER
I20260620 18:32:47.317766 17445 consensus_queue.cc:260] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } }
I20260620 18:32:47.318619 17451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } } }
I20260620 18:32:47.318943 17451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:32:47.319638 17445 sys_catalog.cc:564] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 18:32:47.328265 17462 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:32:47.328435 17462 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:32:47.329922 17378 master_runner.cc:428] Detected that this master 888f00220e5f44c19e0e7f0bff6a1ca9 is joining an existing cluster
I20260620 18:32:47.329996 17378 master_runner.cc:432] Initiating AddMaster RPC to add 127.10.131.61:38067
I20260620 18:32:47.350998 17108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.334272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57904 (local address 127.10.131.62:44649)
0620 18:32:47.334445 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:47.334451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:47.334484 (+ 33us) server_negotiation.cc:407] Connection header received
0620 18:32:47.334540 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:47.334546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:47.334620 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:47.334719 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:47.336368 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.337875 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.340517 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.341420 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.341544 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:47.344731 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:47.344740 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:47.344748 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:47.344798 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:47.347896 (+ 3098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:47.348700 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:47.348708 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:47.348713 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:47.348881 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:47.349317 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:47.349323 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:47.349327 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:47.349664 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:47.349781 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:47.350383 (+ 602us) server_negotiation.cc:299] Negotiation successful
0620 18:32:47.350619 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:32:47.352130 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.333538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.0.0.1:57904)
0620 18:32:47.334237 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:47.334262 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:47.334410 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:47.334771 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:47.334778 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:47.335618 (+ 840us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:47.336142 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:47.336156 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.338097 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.338106 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:47.340313 (+ 2207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:47.340331 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.341909 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.341917 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:47.342193 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:47.342940 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:47.342971 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:47.344437 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:47.348049 (+ 3612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:47.348054 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:47.348058 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:47.348547 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:47.349016 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:47.349020 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:47.349024 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:47.349194 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:47.349870 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:47.349880 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:47.350084 (+ 204us) client_negotiation.cc:769] Sending connection context
0620 18:32:47.351197 (+ 1113us) client_negotiation.cc:240] Negotiation successful
0620 18:32:47.351547 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":9706,"threads_started":1}
I20260620 18:32:47.360497 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.333470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41922 (local address 127.10.131.61:38067)
0620 18:32:47.333703 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:47.333714 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:47.345406 (+ 11692us) server_negotiation.cc:407] Connection header received
0620 18:32:47.345603 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:47.345608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:47.345674 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:47.345734 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:47.347063 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.348223 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.350102 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.350766 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.350856 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:47.353235 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:47.353240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:47.353244 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:47.353274 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:47.357379 (+ 4105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:47.358187 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:47.358192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:47.358195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:47.358330 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:47.358714 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:47.358718 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:47.358721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:47.359065 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:47.359222 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:47.359682 (+ 460us) server_negotiation.cc:299] Negotiation successful
0620 18:32:47.359947 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
WARNING: no policy specified for oryx/127.10.131.60@KRBTEST.COM; defaulting to no policy
I20260620 18:32:47.362162 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.333664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:38067 (local address 127.0.0.1:41922)
0620 18:32:47.345304 (+ 11640us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:47.345330 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:47.345490 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:47.345903 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:47.345908 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:47.346530 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:47.346863 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:47.346879 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.348403 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.348409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:47.349959 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:47.349973 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.351155 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.351162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:47.351319 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:47.351959 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:47.351982 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:47.353048 (+ 1066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:47.357527 (+ 4479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:47.357532 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:47.357535 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:47.358060 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:47.358463 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:47.358466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:47.358469 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:47.358614 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:47.359286 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:47.359289 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:47.359434 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 18:32:47.361214 (+ 1780us) client_negotiation.cc:240] Negotiation successful
0620 18:32:47.361554 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11488,"thread_start_us":11287,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.131.60@KRBTEST.COM" created.
Jun 20 18:32:47 dist-test-slave-cbn3 krb5kdc[16872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980365, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.62@KRBTEST.COM for oryx/127.10.131.61@KRBTEST.COM
I20260620 18:32:47.377977 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.359604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41932 (local address 127.10.131.61:38067)
0620 18:32:47.361177 (+ 1573us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:47.361184 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:47.363070 (+ 1886us) server_negotiation.cc:407] Connection header received
0620 18:32:47.363750 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:47.363756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:47.363872 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:47.363994 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:47.365335 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.366030 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.367461 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.367916 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.368036 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:47.371127 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:47.371132 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:47.371135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:47.371169 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:47.374619 (+ 3450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:47.375535 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:47.375542 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:47.375545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:47.375700 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:47.376133 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:47.376138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:47.376142 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:47.376571 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:47.376722 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:47.377351 (+ 629us) server_negotiation.cc:299] Negotiation successful
0620 18:32:47.377577 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1422,"thread_start_us":3634,"threads_started":1}
I20260620 18:32:47.378167 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.359270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:38067 (local address 127.0.0.1:41932)
0620 18:32:47.362943 (+ 3673us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:47.362991 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:47.363521 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:47.364082 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:47.364090 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:47.364963 (+ 873us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:47.365179 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:47.365189 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.366170 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.366176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:47.367314 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:47.367325 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.368083 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.368089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:47.368272 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:47.368933 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:47.368956 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:47.370949 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:47.374771 (+ 3822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:47.374787 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:47.374798 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:47.375394 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:47.375815 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:47.375823 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:47.375829 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:47.376012 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:47.376725 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:47.376736 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:47.377013 (+ 277us) client_negotiation.cc:769] Sending connection context
0620 18:32:47.377208 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 18:32:47.377600 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3445,"thread_start_us":3277,"threads_started":1}
I20260620 18:32:47.380389 16917 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.10.131.61:38067: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9"
member_type: NON_VOTER
last_known_addr {
host: "127.10.131.61"
port: 38067
}
attrs {
promote: true
}
}
dest_uuid: "2d7e501282a949a1be211a121201e250"
cas_config_opid_index: -1
I20260620 18:32:47.381615 16917 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: NON_VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: true } }
I20260620 18:32:47.386819 17476 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 888f00220e5f44c19e0e7f0bff6a1ca9 (127.10.131.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: NON_VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: true } } }
I20260620 18:32:47.387842 17420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:47.392125 17476 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.10.131.61:38067
I20260620 18:32:47.392455 17420 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [term 1 FOLLOWER]: Refusing update from remote peer 2d7e501282a949a1be211a121201e250: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 18:32:47.393123 17378 master.cc:537] Master@127.10.131.61:38067 shutting down...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.131.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.60.keytab.
Entry for principal oryx/127.10.131.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.60.keytab.
I20260620 18:32:47.397691 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.60:36193
--webserver_interface=127.10.131.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.60.keytab
--principal=oryx/127.10.131.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:37493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:44649,127.10.131.61:38067,127.10.131.60:36193
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:32:47.399274 17477 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [LEADER]: Connected to new peer: Peer: permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: NON_VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:47.401055 17475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2d7e501282a949a1be211a121201e250" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: NON_VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: true } } }
I20260620 18:32:47.402122 17475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [sys.catalog]: This master's current role is: LEADER
W20260620 18:32:47.404518 16904 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260620 18:32:47.405835 17378 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:32:47.406353 17378 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:47.406627 17378 tablet_replica.cc:330] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9: stopping tablet replica
W20260620 18:32:47.410313 16904 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 -> Peer 888f00220e5f44c19e0e7f0bff6a1ca9 (127.10.131.61:38067): Couldn't send request to peer 888f00220e5f44c19e0e7f0bff6a1ca9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 18:32:47.424393 17378 master.cc:552] Master@127.10.131.61:38067 shutdown complete.
I20260620 18:32:47.424552 17378 master_runner.cc:305] Clearing existing system tablet
I20260620 18:32:47.425138 17378 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:47.425256 17378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:47.428341 17378 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 18:32:47.429947 17482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:47.430310 17378 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:47.430565 17378 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:32:47.430686 17378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-1/wal
uuid: "888f00220e5f44c19e0e7f0bff6a1ca9"
format_stamp: "Formatted at 2026-06-20 18:32:47 on dist-test-slave-cbn3"
server_key: "063f761768fc0aae2948d566a36ecb91"
server_key_iv: "6ac329e5143944778f6c0dbdbb3610b1"
server_key_version: "encryptionkey@0"
I20260620 18:32:47.435716 17378 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:47.442396 17378 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 18:32:47.442570 17378 log.cc:1196] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 18:32:47.442965 17378 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9: Deleting consensus metadata
I20260620 18:32:47.443203 17378 master_runner.cc:315] Copying system tablet from 127.10.131.62:44649
I20260620 18:32:47.445779 17378 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9: tablet copy: Beginning tablet copy session from remote peer at address 127.10.131.62:44649
I20260620 18:32:47.462179 17108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.446601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57908 (local address 127.10.131.62:44649)
0620 18:32:47.446787 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:47.446793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:47.447396 (+ 603us) server_negotiation.cc:407] Connection header received
0620 18:32:47.447566 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:47.447572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:47.447669 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:47.447786 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:47.448819 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.450017 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.451775 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.452444 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.452570 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:47.455515 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:47.455522 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:47.455527 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:47.455565 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:47.459033 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:47.459834 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:47.459840 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:47.459844 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:47.459996 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:47.460463 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:47.460468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:47.460472 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:47.460830 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:47.460991 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:47.461575 (+ 584us) server_negotiation.cc:299] Negotiation successful
0620 18:32:47.461819 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:32:47.463456 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.446615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.0.0.1:57908)
0620 18:32:47.447264 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:47.447288 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:47.447444 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:47.447827 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:47.447832 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:47.448344 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:47.448614 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:47.448625 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.450175 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.450181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:47.451620 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:47.451637 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.453011 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.453019 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:47.453293 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:47.454220 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:47.454242 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:47.455289 (+ 1047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:47.459191 (+ 3902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:47.459197 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:47.459203 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:47.459689 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:47.460153 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:47.460158 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:47.460162 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:47.460337 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:47.461086 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:47.461091 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:47.461268 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 18:32:47.462639 (+ 1371us) client_negotiation.cc:240] Negotiation successful
0620 18:32:47.462992 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":332,"threads_started":1}
I20260620 18:32:47.464527 16937 tablet_copy_service.cc:140] P 2d7e501282a949a1be211a121201e250: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 888f00220e5f44c19e0e7f0bff6a1ca9 ({username='oryx', principal='oryx/127.10.131.61@KRBTEST.COM'} at 127.0.0.1:57908)
I20260620 18:32:47.464865 16937 tablet_copy_service.cc:161] P 2d7e501282a949a1be211a121201e250: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 888f00220e5f44c19e0e7f0bff6a1ca9 at {username='oryx', principal='oryx/127.10.131.61@KRBTEST.COM'} at 127.0.0.1:57908: session id = 888f00220e5f44c19e0e7f0bff6a1ca9-00000000000000000000000000000000
I20260620 18:32:47.469437 16937 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 18:32:47.471812 17378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 18:32:47.478389 17378 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9: tablet copy: Starting download of 0 data blocks...
I20260620 18:32:47.478912 17378 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9: tablet copy: Starting download of 1 WAL segments...
I20260620 18:32:47.481647 17378 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 18:32:47.486279 16937 tablet_copy_service.cc:342] P 2d7e501282a949a1be211a121201e250: Request end of tablet copy session 888f00220e5f44c19e0e7f0bff6a1ca9-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.10.131.61@KRBTEST.COM'} at 127.0.0.1:57908
I20260620 18:32:47.486490 16937 tablet_copy_service.cc:433] P 2d7e501282a949a1be211a121201e250: ending tablet copy session 888f00220e5f44c19e0e7f0bff6a1ca9-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 888f00220e5f44c19e0e7f0bff6a1ca9
I20260620 18:32:47.498569 17378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:47.501349 17489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:47.503031 17490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:47.514107 17492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:47.518141 17378 server_base.cc:1034] running on GCE node
I20260620 18:32:47.518515 17378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:47.519263 17378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:47.520584 17378 hybrid_clock.cc:648] HybridClock initialized: now 1781980367520516 us; error 37 us; skew 500 ppm
I20260620 18:32:47.520872 17378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:47 dist-test-slave-cbn3 krb5kdc[16872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980367, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:47.523123 17378 init.cc:377] Logged in from keytab as oryx/127.10.131.61@KRBTEST.COM (short username oryx)
I20260620 18:32:47.525800 17378 webserver.cc:460] Webserver started at http://127.10.131.61:40757/ using document root <none> and password file <none>
I20260620 18:32:47.526175 17378 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:47.526259 17378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:47.529816 17378 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260620 18:32:47.531725 17499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:47.532151 17378 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 18:32:47.532297 17378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-1/wal
uuid: "888f00220e5f44c19e0e7f0bff6a1ca9"
format_stamp: "Formatted at 2026-06-20 18:32:47 on dist-test-slave-cbn3"
server_key: "063f761768fc0aae2948d566a36ecb91"
server_key_iv: "6ac329e5143944778f6c0dbdbb3610b1"
server_key_version: "encryptionkey@0"
I20260620 18:32:47.532465 17378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:47.564028 17378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:47.568348 17378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:47.591079 17378 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.61:38067
I20260620 18:32:47.591099 17550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.61:38067 every 8 connection(s)
I20260620 18:32:47.592165 17378 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-1/data/info.pb
I20260620 18:32:47.599241 17551 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:32:47.603833 17551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9: Bootstrap starting.
I20260620 18:32:47.625806 17551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:32:47.626308 17551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9: Bootstrap complete.
I20260620 18:32:47.627182 17551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: NON_VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: true } }
I20260620 18:32:47.627406 17551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 888f00220e5f44c19e0e7f0bff6a1ca9, State: Initialized, Role: LEARNER
I20260620 18:32:47.627638 17551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: NON_VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: true } }
I20260620 18:32:47.628455 17554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: NON_VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: true } } }
I20260620 18:32:47.628867 17554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [sys.catalog]: This master's current role is: LEARNER
I20260620 18:32:47.629070 17551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:32:47.641571 17565 catalog_manager.cc:1259] Loaded cluster ID: e971777715374237a5b255fc27e8cc4a
I20260620 18:32:47.641707 17565 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9: loading cluster ID for follower catalog manager: success
I20260620 18:32:47.644183 17565 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9: acquiring CA information for follower catalog manager: success
I20260620 18:32:47.646164 17565 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:32:47.783671 17373 heartbeater.cc:503] Master 127.10.131.62:44649 was elected leader, sending a full tablet report...
W20260620 18:32:47.820703 17480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:47.821071 17480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:47.821223 17480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:47.828354 17480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:32:47.828493 17480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:47.828598 17480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:47.828682 17480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:32:47.828758 17480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:32:47.838954 17480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:37493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:44649,127.10.131.61:38067,127.10.131.60:36193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.60.keytab
--principal=oryx/127.10.131.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.60:36193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.131.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:47.840678 17480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:47.842227 17480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:32:47.847252 17569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.840746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41938 (local address 127.10.131.61:38067)
0620 18:32:47.841404 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:47.841408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:47.841424 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:32:47.841484 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:47.841487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:47.841569 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:47.841667 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:47.842157 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.843242 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.845511 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.846344 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.846452 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:47.846616 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:47.846918 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 18:32:47.847015 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":388,"threads_started":1}
I20260620 18:32:47.847311 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.840548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:38067 (local address 127.0.0.1:41938)
0620 18:32:47.840748 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:47.840770 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:47.840928 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:47.841741 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:47.841746 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:47.841776 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:47.841979 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:47.841991 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.843389 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.843395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:47.845354 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:47.845370 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.846533 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.846540 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:47.846733 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:47.846792 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 18:32:47.846895 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 18:32:47.847037 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 18:32:47.848376 17526 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260620 18:32:47.849066 17554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d7e501282a949a1be211a121201e250. Latest consensus state: current_term: 1 leader_uuid: "2d7e501282a949a1be211a121201e250" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: NON_VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: true } } }
I20260620 18:32:47.849416 17554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [sys.catalog]: This master's current role is: LEARNER
W20260620 18:32:47.854866 17571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:47.854928 17572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:47.854969 17574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:47.855697 17480 server_base.cc:1034] running on GCE node
I20260620 18:32:47.856487 17480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:47.857880 17480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:47.859122 17480 hybrid_clock.cc:648] HybridClock initialized: now 1781980367859077 us; error 54 us; skew 500 ppm
I20260620 18:32:47.859459 17480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:47 dist-test-slave-cbn3 krb5kdc[16872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980367, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:47.862854 17480 init.cc:377] Logged in from keytab as oryx/127.10.131.60@KRBTEST.COM (short username oryx)
I20260620 18:32:47.864169 17480 webserver.cc:460] Webserver started at http://127.10.131.60:37289/ using document root <none> and password file <none>
I20260620 18:32:47.864894 17480 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:47.865053 17480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:47.865350 17480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:47.868206 17480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-2/data/instance:
uuid: "1d5bf8ef8b624ecfa578fc93e5492446"
format_stamp: "Formatted at 2026-06-20 18:32:47 on dist-test-slave-cbn3"
server_key: "25caad412fec91544c3afa90d4c34774"
server_key_iv: "39bff71b6a195c2ea7b513ad7e8d056c"
server_key_version: "encryptionkey@0"
I20260620 18:32:47.869060 17480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-2/wal/instance:
uuid: "1d5bf8ef8b624ecfa578fc93e5492446"
format_stamp: "Formatted at 2026-06-20 18:32:47 on dist-test-slave-cbn3"
server_key: "25caad412fec91544c3afa90d4c34774"
server_key_iv: "39bff71b6a195c2ea7b513ad7e8d056c"
server_key_version: "encryptionkey@0"
I20260620 18:32:47.873960 17480 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:32:47.877216 17581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:47.878576 17480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 18:32:47.878746 17480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-2/wal
uuid: "1d5bf8ef8b624ecfa578fc93e5492446"
format_stamp: "Formatted at 2026-06-20 18:32:47 on dist-test-slave-cbn3"
server_key: "25caad412fec91544c3afa90d4c34774"
server_key_iv: "39bff71b6a195c2ea7b513ad7e8d056c"
server_key_version: "encryptionkey@0"
I20260620 18:32:47.878964 17480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:47.907606 17480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:47.911808 17480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:47.912102 17480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:47.930541 17480 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.60:36193
I20260620 18:32:47.930562 17632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.60:36193 every 8 connection(s)
I20260620 18:32:47.931871 17480 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-2/data/info.pb
I20260620 18:32:47.935787 17633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:47.940400 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 17480
I20260620 18:32:47.942341 17633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:32:47 dist-test-slave-cbn3 krb5kdc[16872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980367, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.60@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM
Jun 20 18:32:47 dist-test-slave-cbn3 krb5kdc[16872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.131.60@KRBTEST.COM
I20260620 18:32:47.960039 17108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.945205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57922 (local address 127.10.131.62:44649)
0620 18:32:47.945358 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:47.945362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:47.945965 (+ 603us) server_negotiation.cc:407] Connection header received
0620 18:32:47.946169 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:47.946175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:47.946259 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:47.946361 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:47.947559 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.948317 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.949846 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.950247 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.950379 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:47.953893 (+ 3514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:47.953900 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:47.953905 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:47.953944 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:47.956728 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:47.957633 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:47.957636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:47.957639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:47.957753 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:47.958224 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:47.958228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:47.958230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:47.958488 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:47.958606 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:47.959499 (+ 893us) server_negotiation.cc:299] Negotiation successful
0620 18:32:47.959663 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 18:32:47.960999 17637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.945193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.0.0.1:57922)
0620 18:32:47.945816 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:47.945855 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:47.946036 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:47.946438 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:47.946444 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:47.947041 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:47.947391 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:47.947407 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.948469 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.948474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:47.949706 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:47.949717 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.950469 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.950476 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:47.950647 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:47.951390 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:47.951411 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:47.953702 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:47.956881 (+ 3179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:47.956892 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:47.956906 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:47.957519 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:47.957864 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:47.957869 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:47.957873 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:47.958112 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:47.958604 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:47.958614 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:47.959036 (+ 422us) client_negotiation.cc:769] Sending connection context
0620 18:32:47.959322 (+ 286us) client_negotiation.cc:240] Negotiation successful
0620 18:32:47.959817 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":50,"thread_start_us":316,"threads_started":1}
I20260620 18:32:47.961549 17636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.942526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52170 (local address 127.10.131.60:36193)
0620 18:32:47.943276 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:47.943288 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:47.943325 (+ 37us) server_negotiation.cc:407] Connection header received
0620 18:32:47.945319 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:47.945352 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:47.945881 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:47.946213 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:47.947659 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.948779 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.950025 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.950533 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.950672 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:47.955119 (+ 4447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:47.955127 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:47.955139 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:47.955166 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:47.958338 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:47.959115 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:47.959121 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:47.959123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:47.959327 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:47.959732 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:47.959737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:47.959740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:47.960173 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:47.960487 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:47.960906 (+ 419us) server_negotiation.cc:299] Negotiation successful
0620 18:32:47.961150 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":552,"thread_start_us":339,"threads_started":1}
I20260620 18:32:47.962611 17633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:32:47 dist-test-slave-cbn3 krb5kdc[16872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980367, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.60@KRBTEST.COM for oryx/127.10.131.61@KRBTEST.COM
I20260620 18:32:47.974504 17569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.963360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41940 (local address 127.10.131.61:38067)
0620 18:32:47.963551 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:47.963558 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:47.963763 (+ 205us) server_negotiation.cc:407] Connection header received
0620 18:32:47.963957 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:47.963964 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:47.964031 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:47.964108 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:47.964705 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.965438 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.966613 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.966974 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.967060 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:47.969810 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:47.969814 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:47.969818 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:47.969847 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:47.972053 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:47.972672 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:47.972682 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:47.972685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:47.972804 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:47.973200 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:47.973206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:47.973210 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:47.973479 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:47.973623 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:47.973964 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 18:32:47.974180 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:32:47.974498 17637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.963387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:38067 (local address 127.0.0.1:41940)
0620 18:32:47.963685 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:47.963703 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:47.963834 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:47.964139 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:47.964142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:47.964468 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:47.964593 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:47.964600 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.965564 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.965568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:47.966497 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:47.966507 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.967082 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.967086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:47.967216 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:47.967789 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:47.967804 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:47.969629 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:47.972180 (+ 2551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:47.972187 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:47.972191 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:47.972553 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:47.972965 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:47.972970 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:47.972973 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:47.973099 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:47.973588 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:47.973593 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:47.973732 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 18:32:47.973870 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 18:32:47.974142 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260620 18:32:47.975603 17633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36193 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:32:47 dist-test-slave-cbn3 krb5kdc[16872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980367, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.60@KRBTEST.COM for oryx/127.10.131.60@KRBTEST.COM
I20260620 18:32:47.987118 17637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.976330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:36193 (local address 127.0.0.1:52186)
0620 18:32:47.976471 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:47.976489 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:47.976576 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:47.976805 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:47.976808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:47.977151 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:47.977284 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:47.977291 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.978217 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.978220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:47.979021 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:47.979030 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.979649 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.979653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:47.979798 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:47.980251 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:47.980267 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:47.982110 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:47.985220 (+ 3110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:47.985223 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:47.985226 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:47.985565 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:47.985846 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:47.985849 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:47.985851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:47.985983 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:47.986418 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:47.986421 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:47.986524 (+ 103us) client_negotiation.cc:769] Sending connection context
0620 18:32:47.986653 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 18:32:47.986822 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 18:32:47.987151 17636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:47.976283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52186 (local address 127.10.131.60:36193)
0620 18:32:47.976425 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:47.976429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:47.976548 (+ 119us) server_negotiation.cc:407] Connection header received
0620 18:32:47.976653 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:47.976656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:47.976704 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:47.976778 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:47.977440 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.978109 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.979141 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:47.979536 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:47.979631 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:47.982299 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:47.982304 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:47.982307 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:47.982332 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:47.985120 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:47.985653 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:47.985656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:47.985658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:47.985761 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:47.986063 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:47.986065 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:47.986067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:47.986332 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:47.986430 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:47.986733 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 18:32:47.986924 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":48}
I20260620 18:32:47.991789 17633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446: Bootstrap starting.
I20260620 18:32:47.994932 17633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:47.995930 17633 log.cc:824] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:47.998601 17633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446: No bootstrap required, opened a new log
I20260620 18:32:48.005573 17633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } } peers { permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36193 } }
I20260620 18:32:48.005892 17633 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:48.006018 17633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d5bf8ef8b624ecfa578fc93e5492446, State: Initialized, Role: FOLLOWER
I20260620 18:32:48.006549 17633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } } peers { permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36193 } }
I20260620 18:32:48.007406 17639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } } peers { permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36193 } } }
I20260620 18:32:48.007817 17639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:32:48.008455 17633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:32:48.016058 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 16978
W20260620 18:32:48.017407 17650 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:32:48.017624 17650 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:32:48.021929 17480 master_runner.cc:428] Detected that this master 1d5bf8ef8b624ecfa578fc93e5492446 is joining an existing cluster
I20260620 18:32:48.022053 17480 master_runner.cc:432] Initiating AddMaster RPC to add 127.10.131.60:36193
Jun 20 18:32:48 dist-test-slave-cbn3 krb5kdc[16872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.131.1@KRBTEST.COM
I20260620 18:32:48.038396 17652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:48.017935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37042 (local address 127.10.131.1:45237)
0620 18:32:48.018555 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:48.018566 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:48.018596 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:32:48.018668 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:48.018677 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:48.018922 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:48.019104 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:48.020695 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.021962 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.023941 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.024825 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.024960 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:48.030972 (+ 6012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:48.030988 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:48.031006 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:48.031051 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:48.035038 (+ 3987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:48.035882 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:48.035893 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:48.035896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:48.036034 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:48.036527 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:48.036532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:48.036535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:48.036927 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:48.037203 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:48.037742 (+ 539us) server_negotiation.cc:299] Negotiation successful
0620 18:32:48.038033 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":288,"threads_started":1}
I20260620 18:32:48.039793 17101 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:48.044116 17108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:48.026975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57936 (local address 127.10.131.62:44649)
0620 18:32:48.027237 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:48.027244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:48.027263 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:32:48.027322 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:48.027327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:48.027418 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:48.027505 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:48.028888 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.030106 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.032269 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.032977 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.033109 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:48.035667 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:48.035673 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:48.035677 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:48.035716 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:48.040389 (+ 4673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:48.041295 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:48.041304 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:48.041310 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:48.041551 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:48.042146 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:48.042154 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:48.042160 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:48.042601 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:48.042759 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:48.043524 (+ 765us) server_negotiation.cc:299] Negotiation successful
0620 18:32:48.043741 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260620 18:32:48.045727 17657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:48.025875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.0.0.1:57936)
0620 18:32:48.026514 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:48.026538 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:48.026658 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:48.027519 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:48.027524 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:48.028244 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:48.028686 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:48.028698 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.030340 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.030345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:48.032050 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:48.032076 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.033235 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.033246 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:48.033504 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:48.034163 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:48.034185 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:48.035450 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:48.040545 (+ 5095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:48.040550 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:48.040554 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:48.041170 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:48.041730 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:48.041738 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:48.041745 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:48.041978 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:48.042836 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:48.042841 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:48.043074 (+ 233us) client_negotiation.cc:769] Sending connection context
0620 18:32:48.043227 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 18:32:48.045134 (+ 1907us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":23,"thread_start_us":1915,"threads_started":1}
I20260620 18:32:48.046768 17639 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:32:48.047098 17639 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } } peers { permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36193 } }
I20260620 18:32:48.049499 16927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d7e501282a949a1be211a121201e250" is_pre_election: true
I20260620 18:32:48.050560 17636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:48.027724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52190 (local address 127.10.131.60:36193)
0620 18:32:48.027912 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:48.027919 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:48.027939 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:32:48.027994 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:48.027999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:48.028074 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:48.028189 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:48.030524 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.031688 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.034383 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.035168 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.035291 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:48.043162 (+ 7871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:48.043171 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:48.043178 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:48.043217 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:48.047469 (+ 4252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:48.048190 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:48.048198 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:48.048201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:48.048380 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:48.048797 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:48.048803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:48.048806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:48.049232 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:48.049394 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:48.049911 (+ 517us) server_negotiation.cc:299] Negotiation successful
0620 18:32:48.050145 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":43,"spinlock_wait_cycles":25344}
I20260620 18:32:48.050796 17526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" is_pre_election: true
I20260620 18:32:48.051553 17658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:48.026588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:36193 (local address 127.0.0.1:52190)
0620 18:32:48.027218 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:48.027239 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:48.027337 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:48.028419 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:48.028425 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:48.030119 (+ 1694us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:48.030384 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:48.030392 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.032952 (+ 2560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.032958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:48.034258 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:48.034274 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.035616 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.035622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:48.035791 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:48.041382 (+ 5591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:48.041411 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:48.042942 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:48.047585 (+ 4643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:48.047590 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:48.047594 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:48.048072 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:48.048518 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:48.048522 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:48.048525 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:48.048675 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:48.049459 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:48.049466 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:48.049637 (+ 171us) client_negotiation.cc:769] Sending connection context
0620 18:32:48.050771 (+ 1134us) client_negotiation.cc:240] Negotiation successful
0620 18:32:48.051082 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":1437,"threads_started":1}
I20260620 18:32:48.048779 17639 leader_election.cc:288] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d7e501282a949a1be211a121201e250 (127.10.131.62:44649), 888f00220e5f44c19e0e7f0bff6a1ca9 (127.10.131.61:38067)
I20260620 18:32:48.052440 17585 leader_election.cc:302] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d5bf8ef8b624ecfa578fc93e5492446; no voters: 2d7e501282a949a1be211a121201e250, 888f00220e5f44c19e0e7f0bff6a1ca9
I20260620 18:32:48.052904 17639 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Jun 20 18:32:48 dist-test-slave-cbn3 krb5kdc[16872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980365, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.62@KRBTEST.COM for oryx/127.10.131.60@KRBTEST.COM
I20260620 18:32:48.071143 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:48.054469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:36193 (local address 127.0.0.1:52198)
0620 18:32:48.054666 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:48.054686 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:48.054800 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:48.055549 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:48.055554 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:48.055998 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:48.056201 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:48.056213 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.057824 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.057831 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:48.059494 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:48.059509 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.060642 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.060648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:48.060828 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:48.061482 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:48.061504 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:48.064231 (+ 2727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:48.068206 (+ 3975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:48.068211 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:48.068216 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:48.068708 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:48.069130 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:48.069135 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:48.069139 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:48.069276 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:48.069871 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:48.069876 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:48.070034 (+ 158us) client_negotiation.cc:769] Sending connection context
0620 18:32:48.070164 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 18:32:48.070483 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 18:32:48.071259 17659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:48.030738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:38067 (local address 127.0.0.1:41944)
0620 18:32:48.045839 (+ 15101us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:48.045870 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:48.046295 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:48.046865 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:48.046870 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:48.057049 (+ 10179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:48.057288 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:48.057296 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.058700 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.058717 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:48.060543 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:48.060559 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.062132 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.062139 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:48.062326 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:48.062951 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:48.062980 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:48.064025 (+ 1045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:48.068311 (+ 4286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:48.068316 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:48.068319 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:48.068824 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:48.069292 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:48.069297 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:48.069301 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:48.069477 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:48.070181 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:48.070186 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:48.070368 (+ 182us) client_negotiation.cc:769] Sending connection context
0620 18:32:48.070507 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 18:32:48.070800 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14936,"mutex_wait_us":3,"thread_start_us":15240,"threads_started":1}
I20260620 18:32:48.072163 17569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:48.028665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41944 (local address 127.10.131.61:38067)
0620 18:32:48.028888 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:48.028894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:48.045963 (+ 17069us) server_negotiation.cc:407] Connection header received
0620 18:32:48.046470 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:48.046476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:48.046566 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:48.046638 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:48.057440 (+ 10802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.058519 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.060733 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.061763 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.061893 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:48.064240 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:48.064246 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:48.064251 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:48.064299 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:48.068161 (+ 3862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:48.068984 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:48.068989 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:48.068993 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:48.069156 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:48.069603 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:48.069608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:48.069612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:48.069946 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:48.070088 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:48.071576 (+ 1488us) server_negotiation.cc:299] Negotiation successful
0620 18:32:48.071834 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:32:48.072539 17636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:48.054995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52198 (local address 127.10.131.60:36193)
0620 18:32:48.055160 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:48.055166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:48.055183 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:32:48.055295 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:48.055301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:48.055390 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:48.055494 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:48.056366 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.057638 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.059671 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.060461 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.061149 (+ 688us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:48.064438 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:48.064444 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:48.064449 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:48.064482 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:48.068067 (+ 3585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:48.068882 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:48.068887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:48.068891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:48.069014 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:48.069415 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:48.069420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:48.069423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:48.069744 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:48.071890 (+ 2146us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:48.072018 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 18:32:48.072215 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":54}
I20260620 18:32:48.074529 16917 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.10.131.60:36193: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446"
member_type: NON_VOTER
last_known_addr {
host: "127.10.131.60"
port: 36193
}
attrs {
promote: true
}
}
dest_uuid: "2d7e501282a949a1be211a121201e250"
cas_config_opid_index: 5
I20260620 18:32:48.075593 16917 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: NON_VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: true } } peers { permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: NON_VOTER last_known_addr { host: "127.10.131.60" port: 36193 } attrs { promote: true } }
I20260620 18:32:48.077332 17475 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1d5bf8ef8b624ecfa578fc93e5492446 (127.10.131.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: NON_VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: true } } peers { permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: NON_VOTER last_known_addr { host: "127.10.131.60" port: 36193 } attrs { promote: true } } }
I20260620 18:32:48.077968 17608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:48.080612 17608 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [term 1 FOLLOWER]: Refusing update from remote peer 2d7e501282a949a1be211a121201e250: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 18:32:48.100442 17526 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [term 1 LEARNER]: Refusing update from remote peer 2d7e501282a949a1be211a121201e250: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 18:32:48.106941 17475 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.10.131.60:36193
I20260620 18:32:48.107033 17664 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: NON_VOTER last_known_addr { host: "127.10.131.60" port: 36193 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:48.107748 17480 master.cc:537] Master@127.10.131.60:36193 shutting down...
I20260620 18:32:48.108800 17661 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [LEADER]: Connected to new peer: Peer: permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: NON_VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 18:32:48.116747 17663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2d7e501282a949a1be211a121201e250" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: NON_VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: true } } peers { permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: NON_VOTER last_known_addr { host: "127.10.131.60" port: 36193 } attrs { promote: true } } }
I20260620 18:32:48.117956 17663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:48.122551 17526 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1d5bf8ef8b624ecfa578fc93e5492446 (127.10.131.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: NON_VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: true } } peers { permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: NON_VOTER last_known_addr { host: "127.10.131.60" port: 36193 } attrs { promote: true } } }
I20260620 18:32:48.127841 17554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2d7e501282a949a1be211a121201e250" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: NON_VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: true } } peers { permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: NON_VOTER last_known_addr { host: "127.10.131.60" port: 36193 } attrs { promote: true } } }
I20260620 18:32:48.128127 17554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [sys.catalog]: This master's current role is: LEARNER
I20260620 18:32:48.129237 17661 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250: attempting to promote NON_VOTER 888f00220e5f44c19e0e7f0bff6a1ca9 to VOTER
I20260620 18:32:48.132148 17661 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: false } } peers { permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: NON_VOTER last_known_addr { host: "127.10.131.60" port: 36193 } attrs { promote: true } }
W20260620 18:32:48.132962 16901 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.131.60:36193, user_credentials={real_user=slave}} blocked reactor thread for 51611.3us
I20260620 18:32:48.134763 17526 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [term 1 LEARNER]: Refusing update from remote peer 2d7e501282a949a1be211a121201e250: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 18:32:48.135468 17661 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [LEADER]: Connected to new peer: Peer: permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 18:32:48.138732 17664 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 888f00220e5f44c19e0e7f0bff6a1ca9 (127.10.131.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: false } } peers { permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: NON_VOTER last_known_addr { host: "127.10.131.60" port: 36193 } attrs { promote: true } } }
I20260620 18:32:48.143842 17480 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:32:48.144251 17480 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:48.144443 17480 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446: stopping tablet replica
I20260620 18:32:48.145241 17664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2d7e501282a949a1be211a121201e250" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: false } } peers { permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: NON_VOTER last_known_addr { host: "127.10.131.60" port: 36193 } attrs { promote: true } } }
I20260620 18:32:48.145490 17664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:48.145715 17664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2d7e501282a949a1be211a121201e250" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: false } } peers { permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: NON_VOTER last_known_addr { host: "127.10.131.60" port: 36193 } attrs { promote: true } } }
I20260620 18:32:48.145911 17664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:48.151773 17526 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 888f00220e5f44c19e0e7f0bff6a1ca9 (127.10.131.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: false } } peers { permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: NON_VOTER last_known_addr { host: "127.10.131.60" port: 36193 } attrs { promote: true } } }
I20260620 18:32:48.152580 17480 master.cc:552] Master@127.10.131.60:36193 shutdown complete.
I20260620 18:32:48.152704 17480 master_runner.cc:305] Clearing existing system tablet
I20260620 18:32:48.153142 17480 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:48.153293 17480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:48.155915 17554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2d7e501282a949a1be211a121201e250" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: false } } peers { permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: NON_VOTER last_known_addr { host: "127.10.131.60" port: 36193 } attrs { promote: true } } }
I20260620 18:32:48.156190 17554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:32:48.161497 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:48.161065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:36193 (local address 127.0.0.1:52210)
0620 18:32:48.161211 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:48.161330 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 18:32:48.161762 16901 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 -> Peer 1d5bf8ef8b624ecfa578fc93e5492446 (127.10.131.60:36193): Couldn't send request to peer 1d5bf8ef8b624ecfa578fc93e5492446. Status: Network error: Client connection negotiation failed: client connection to 127.10.131.60:36193: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 18:32:48.161851 17480 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.002s sys 0.000s
I20260620 18:32:48.173256 17669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:48.173650 17480 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:48.173916 17480 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.003s sys 0.007s
I20260620 18:32:48.174017 17480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-2/wal
uuid: "1d5bf8ef8b624ecfa578fc93e5492446"
format_stamp: "Formatted at 2026-06-20 18:32:47 on dist-test-slave-cbn3"
server_key: "25caad412fec91544c3afa90d4c34774"
server_key_iv: "39bff71b6a195c2ea7b513ad7e8d056c"
server_key_version: "encryptionkey@0"
I20260620 18:32:48.179168 17480 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:48.186791 17480 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 18:32:48.186960 17480 log.cc:1196] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 18:32:48.187518 17480 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446: Deleting consensus metadata
I20260620 18:32:48.187788 17480 master_runner.cc:315] Copying system tablet from 127.10.131.62:44649
I20260620 18:32:48.187850 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 16978
I20260620 18:32:48.197441 17480 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446: tablet copy: Beginning tablet copy session from remote peer at address 127.10.131.62:44649
I20260620 18:32:48.207995 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.1:45237
--local_ip_for_outbound_sockets=127.10.131.1
--tserver_master_addrs=127.10.131.62:44649
--webserver_port=44583
--webserver_interface=127.10.131.1
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.1.keytab
--principal=oryx/127.10.131.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:37493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.131.62:44649,127.10.131.61:38067
--tserver_master_addrs=127.10.131.62:44649,127.10.131.61:38067,127.10.131.60:36193 with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:32:48.214356 17108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:48.198319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57950 (local address 127.10.131.62:44649)
0620 18:32:48.198558 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:48.198564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:48.199105 (+ 541us) server_negotiation.cc:407] Connection header received
0620 18:32:48.199300 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:48.199309 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:48.199400 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:48.199511 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:48.200499 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.201923 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.203892 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.204736 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.204917 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:48.207371 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:48.207378 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:48.207383 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:48.207425 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:48.211065 (+ 3640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:48.211940 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:48.211946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:48.211950 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:48.212098 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:48.212541 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:48.212546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:48.212550 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:48.212924 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:48.213066 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:48.213648 (+ 582us) server_negotiation.cc:299] Negotiation successful
0620 18:32:48.213905 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260620 18:32:48.214352 17675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:48.198270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.0.0.1:57950)
0620 18:32:48.198982 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:48.199003 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:48.199148 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:48.199553 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:48.199558 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:48.200071 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:48.200332 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:48.200344 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.202071 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.202077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:48.203737 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:48.203753 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.204936 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.204942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:48.205154 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:48.205932 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:48.205962 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:48.207128 (+ 1166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:48.211268 (+ 4140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:48.211273 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:48.211278 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:48.211789 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:48.212239 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:48.212244 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:48.212248 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:48.212411 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:48.213123 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:48.213129 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:48.213344 (+ 215us) client_negotiation.cc:769] Sending connection context
0620 18:32:48.213501 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 18:32:48.213829 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":400,"threads_started":1}
I20260620 18:32:48.215025 16937 tablet_copy_service.cc:140] P 2d7e501282a949a1be211a121201e250: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1d5bf8ef8b624ecfa578fc93e5492446 ({username='oryx', principal='oryx/127.10.131.60@KRBTEST.COM'} at 127.0.0.1:57950)
I20260620 18:32:48.215202 16937 tablet_copy_service.cc:161] P 2d7e501282a949a1be211a121201e250: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1d5bf8ef8b624ecfa578fc93e5492446 at {username='oryx', principal='oryx/127.10.131.60@KRBTEST.COM'} at 127.0.0.1:57950: session id = 1d5bf8ef8b624ecfa578fc93e5492446-00000000000000000000000000000000
I20260620 18:32:48.217962 16937 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 18:32:48.219872 17480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 18:32:48.227962 17480 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446: tablet copy: Starting download of 0 data blocks...
I20260620 18:32:48.228323 17480 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446: tablet copy: Starting download of 1 WAL segments...
I20260620 18:32:48.231077 17480 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 18:32:48.235893 16937 tablet_copy_service.cc:342] P 2d7e501282a949a1be211a121201e250: Request end of tablet copy session 1d5bf8ef8b624ecfa578fc93e5492446-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.10.131.60@KRBTEST.COM'} at 127.0.0.1:57950
I20260620 18:32:48.236037 16937 tablet_copy_service.cc:433] P 2d7e501282a949a1be211a121201e250: ending tablet copy session 1d5bf8ef8b624ecfa578fc93e5492446-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1d5bf8ef8b624ecfa578fc93e5492446
I20260620 18:32:48.246081 17480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:48.248719 17677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:48.255934 17678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:48.263000 17480 server_base.cc:1034] running on GCE node
W20260620 18:32:48.263767 17680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:48.264200 17480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:48.264950 17480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:48.266187 17480 hybrid_clock.cc:648] HybridClock initialized: now 1781980368266144 us; error 36 us; skew 500 ppm
I20260620 18:32:48.266417 17480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:48 dist-test-slave-cbn3 krb5kdc[16872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980368, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:48.268222 17480 init.cc:377] Logged in from keytab as oryx/127.10.131.60@KRBTEST.COM (short username oryx)
I20260620 18:32:48.272931 17480 webserver.cc:460] Webserver started at http://127.10.131.60:39487/ using document root <none> and password file <none>
I20260620 18:32:48.273263 17480 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:48.273370 17480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:48.275707 17480 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 18:32:48.277370 17687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:48.277756 17480 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:32:48.277894 17480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-2/wal
uuid: "1d5bf8ef8b624ecfa578fc93e5492446"
format_stamp: "Formatted at 2026-06-20 18:32:47 on dist-test-slave-cbn3"
server_key: "25caad412fec91544c3afa90d4c34774"
server_key_iv: "39bff71b6a195c2ea7b513ad7e8d056c"
server_key_version: "encryptionkey@0"
I20260620 18:32:48.278087 17480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:48.288357 17480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:48.292140 17480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:48.319932 17480 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.60:36193
I20260620 18:32:48.319958 17738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.60:36193 every 8 connection(s)
I20260620 18:32:48.320890 17480 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/master-2/data/info.pb
I20260620 18:32:48.327425 17739 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:32:48.332584 17739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446: Bootstrap starting.
I20260620 18:32:48.349030 17739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:32:48.349493 17739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446: Bootstrap complete.
I20260620 18:32:48.350312 17739 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: false } } peers { permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: NON_VOTER last_known_addr { host: "127.10.131.60" port: 36193 } attrs { promote: true } }
I20260620 18:32:48.350548 17739 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1d5bf8ef8b624ecfa578fc93e5492446, State: Initialized, Role: LEARNER
I20260620 18:32:48.350768 17739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: false } } peers { permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: NON_VOTER last_known_addr { host: "127.10.131.60" port: 36193 } attrs { promote: true } }
I20260620 18:32:48.353308 17742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: false } } peers { permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: NON_VOTER last_known_addr { host: "127.10.131.60" port: 36193 } attrs { promote: true } } }
I20260620 18:32:48.353645 17739 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:32:48.353751 17742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [sys.catalog]: This master's current role is: LEARNER
I20260620 18:32:48.365592 17753 catalog_manager.cc:1259] Loaded cluster ID: e971777715374237a5b255fc27e8cc4a
I20260620 18:32:48.365712 17753 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446: loading cluster ID for follower catalog manager: success
I20260620 18:32:48.368052 17753 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446: acquiring CA information for follower catalog manager: success
I20260620 18:32:48.369903 17753 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:32:48.542595 17754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:48.536745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50892 (local address 127.10.131.60:36193)
0620 18:32:48.537318 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:48.537322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:48.537340 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:32:48.537398 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:48.537402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:48.537464 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:48.537592 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:48.537990 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.539235 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.540723 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.541583 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.541725 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:48.541972 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:48.542132 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 18:32:48.542259 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":338,"threads_started":1}
I20260620 18:32:48.542572 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:48.536556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:36193 (local address 127.0.0.1:50892)
0620 18:32:48.536714 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:48.536738 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:48.536884 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:48.537617 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:48.537622 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:48.537670 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:48.537841 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:48.537849 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.539389 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.539395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:48.540578 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:48.540590 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.541756 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.541761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:48.541951 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:48.541988 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 18:32:48.542109 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 18:32:48.542248 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 18:32:48.543751 17714 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 18:32:48.544318 17742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d7e501282a949a1be211a121201e250. Latest consensus state: current_term: 1 leader_uuid: "2d7e501282a949a1be211a121201e250" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: false } } peers { permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: NON_VOTER last_known_addr { host: "127.10.131.60" port: 36193 } attrs { promote: true } } }
I20260620 18:32:48.544628 17742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [sys.catalog]: This master's current role is: LEARNER
W20260620 18:32:48.601068 17676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:48.601389 17676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:48.601543 17676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:48.609192 17676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:48.609335 17676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:48.609504 17676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.1
I20260620 18:32:48.619278 17676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:37493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.1.keytab
--principal=oryx/127.10.131.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.1:45237
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.131.1
--webserver_port=44583
--tserver_master_addrs=127.10.131.62:44649,127.10.131.61:38067,127.10.131.60:36193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.1
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:48.620658 17676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:48.621981 17676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:48.635161 17760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:48.635121 17759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:48.635170 17762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:48.635802 17676 server_base.cc:1034] running on GCE node
I20260620 18:32:48.636400 17676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:48.637391 17676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:48.638610 17676 hybrid_clock.cc:648] HybridClock initialized: now 1781980368638566 us; error 37 us; skew 500 ppm
I20260620 18:32:48.638931 17676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:48 dist-test-slave-cbn3 krb5kdc[16872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980368, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:48.642059 17676 init.cc:377] Logged in from keytab as oryx/127.10.131.1@KRBTEST.COM (short username oryx)
I20260620 18:32:48.643642 17676 webserver.cc:460] Webserver started at http://127.10.131.1:44583/ using document root <none> and password file <none>
I20260620 18:32:48.644326 17676 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:48.644484 17676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:48.649890 17676 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:32:48.652639 17769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:48.654095 17676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:32:48.654258 17676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-0/wal
uuid: "2694a3ddafbc484e882015fb46f9bbfc"
format_stamp: "Formatted at 2026-06-20 18:32:45 on dist-test-slave-cbn3"
server_key: "fcf2359cb5bca6a0d00eee86e1c57bc3"
server_key_iv: "de625206061749554c6164b817804e0b"
server_key_version: "encryptionkey@0"
I20260620 18:32:48.655205 17676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:48.677444 17676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:48.681959 17676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:48.682246 17676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:48.683297 17676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:48.684926 17676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:48.685073 17676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 18:32:48.685230 17676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:48.685344 17676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:48.719064 17676 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.1:45237
I20260620 18:32:48.719153 17881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.1:45237 every 8 connection(s)
I20260620 18:32:48.720563 17676 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
I20260620 18:32:48.722802 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 17676
I20260620 18:32:48.723176 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 17112
Jun 20 18:32:48 dist-test-slave-cbn3 krb5kdc[16872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980368, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.60@KRBTEST.COM
Jun 20 18:32:48 dist-test-slave-cbn3 krb5kdc[16872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.131.2@KRBTEST.COM
Jun 20 18:32:48 dist-test-slave-cbn3 krb5kdc[16872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980368, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM
Jun 20 18:32:48 dist-test-slave-cbn3 krb5kdc[16872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980368, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.61@KRBTEST.COM
I20260620 18:32:48.760167 17754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:48.725159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:59351 (local address 127.10.131.60:36193)
0620 18:32:48.725368 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:48.725373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:48.739368 (+ 13995us) server_negotiation.cc:407] Connection header received
0620 18:32:48.739536 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:48.739540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:48.739606 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:48.739743 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:48.741882 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.742584 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.744804 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.745678 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.745826 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:48.752351 (+ 6525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:48.752360 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:48.752365 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:48.752408 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:48.755728 (+ 3320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:48.756772 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:48.756779 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:48.756783 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:48.756957 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:48.757563 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:48.757571 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:48.757578 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:48.758044 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:48.758198 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:48.759441 (+ 1243us) server_negotiation.cc:299] Negotiation successful
0620 18:32:48.759680 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:32:48.761713 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:48.726035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:36193 (local address 127.10.131.1:59351)
0620 18:32:48.739228 (+ 13193us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:48.739260 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:48.739413 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:48.740035 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:48.740044 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:48.740741 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:48.741668 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:48.741690 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.742752 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.742758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:48.744599 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:48.744615 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.745922 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.745930 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:48.746280 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:48.748159 (+ 1879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:48.748206 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:48.752040 (+ 3834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:48.755905 (+ 3865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:48.755916 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:48.755934 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:48.756629 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:48.757151 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:48.757156 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:48.757159 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:48.757405 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:48.758337 (+ 932us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:48.758347 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:48.758884 (+ 537us) client_negotiation.cc:769] Sending connection context
0620 18:32:48.759301 (+ 417us) client_negotiation.cc:240] Negotiation successful
0620 18:32:48.760008 (+ 707us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13053,"thread_start_us":1002,"threads_started":1}
I20260620 18:32:48.761742 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:48.724616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.10.131.1:43243)
0620 18:32:48.727584 (+ 2968us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:48.727645 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:48.731137 (+ 3492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:48.746188 (+ 15051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:48.746193 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:48.746712 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:48.747041 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:48.747055 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.748322 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.748328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:48.750090 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:48.750105 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.751165 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.751172 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:48.751369 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:48.752092 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:48.752115 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:48.755022 (+ 2907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:48.758858 (+ 3836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:48.758865 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:48.758871 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:48.759374 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:48.759796 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:48.759814 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:48.759820 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:48.759984 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:48.760685 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:48.760689 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:48.760836 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 18:32:48.761001 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 18:32:48.761271 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2563,"mutex_wait_us":37,"thread_start_us":20271,"threads_started":1}
I20260620 18:32:48.763504 17882 heartbeater.cc:347] Connected to a master server at 127.10.131.60:36193
I20260620 18:32:48.763535 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:48.725616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:43243 (local address 127.10.131.62:44649)
0620 18:32:48.745839 (+ 20223us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:48.745845 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:48.745865 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:32:48.745922 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:48.745929 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:48.746006 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:48.747213 (+ 1207us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:48.747378 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.748161 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.750277 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.750941 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.751716 (+ 775us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:48.755257 (+ 3541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:48.755266 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:48.755271 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:48.755312 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:48.758694 (+ 3382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:48.759527 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:48.759531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:48.759534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:48.759671 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:48.760122 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:48.760127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:48.760130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:48.760458 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:48.762618 (+ 2160us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:48.762760 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 18:32:48.763140 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20091,"thread_start_us":19995,"threads_started":1}
I20260620 18:32:48.763859 17882 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:48.764297 17883 heartbeater.cc:347] Connected to a master server at 127.10.131.62:44649
I20260620 18:32:48.764457 17883 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:48.764935 17882 heartbeater.cc:511] Master 127.10.131.60:36193 requested a full tablet report, sending...
I20260620 18:32:48.764987 17883 heartbeater.cc:511] Master 127.10.131.62:44649 requested a full tablet report, sending...
I20260620 18:32:48.765761 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:48.725855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38294 (local address 127.10.131.2:35827)
0620 18:32:48.737597 (+ 11742us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:48.737611 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:48.737660 (+ 49us) server_negotiation.cc:407] Connection header received
0620 18:32:48.737763 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:48.737776 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:48.738188 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:48.738431 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:48.741288 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.743410 (+ 2122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.745429 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.746327 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.747058 (+ 731us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:48.755394 (+ 8336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:48.755412 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:48.755432 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:48.755492 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:48.760664 (+ 5172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:48.761715 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:48.761725 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:48.761732 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:48.761892 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:48.762392 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:48.762397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:48.762401 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:48.763143 (+ 742us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:48.763441 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:48.765027 (+ 1586us) server_negotiation.cc:299] Negotiation successful
0620 18:32:48.765313 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11472,"thread_start_us":11285,"threads_started":1}
I20260620 18:32:48.766779 16917 ts_manager.cc:194] Re-registered known tserver with Master: 2694a3ddafbc484e882015fb46f9bbfc (127.10.131.1:45237)
I20260620 18:32:48.767105 17235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:48.767338 17704 ts_manager.cc:194] Registered new tserver with Master: 2694a3ddafbc484e882015fb46f9bbfc (127.10.131.1:45237)
I20260620 18:32:48.768079 16917 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.131.1@KRBTEST.COM'} at 127.10.131.1:43243
I20260620 18:32:48.770933 17886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:48.725264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:43005 (local address 127.10.131.61:38067)
0620 18:32:48.725791 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:48.725797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:48.745476 (+ 19679us) server_negotiation.cc:407] Connection header received
0620 18:32:48.746282 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:48.746288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:48.746385 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:48.746491 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:48.748024 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.749208 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.751538 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.752375 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.752486 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:48.761464 (+ 8978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:48.761472 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:48.761477 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:48.761518 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:48.767720 (+ 6202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:48.768551 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:48.768560 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:48.768566 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:48.768749 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:48.769261 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:48.769267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:48.769271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:48.769627 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:48.769805 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:48.770286 (+ 481us) server_negotiation.cc:299] Negotiation successful
0620 18:32:48.770516 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":502,"threads_started":1}
I20260620 18:32:48.770968 17889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:48.726074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:38067 (local address 127.10.131.1:43005)
0620 18:32:48.745363 (+ 19289us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:48.745392 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:48.746092 (+ 700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:48.746791 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:48.746797 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:48.747497 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:48.747832 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:48.747847 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.749392 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.749397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:48.751351 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:48.751364 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:48.752768 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:48.752775 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:48.752989 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:48.757536 (+ 4547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:48.757558 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:48.761231 (+ 3673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:48.767893 (+ 6662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:48.767900 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:48.767903 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:48.768408 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:48.768926 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:48.768931 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:48.768935 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:48.769132 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:48.769767 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:48.769773 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:48.770012 (+ 239us) client_negotiation.cc:769] Sending connection context
0620 18:32:48.770165 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 18:32:48.770476 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19138,"mutex_wait_us":91,"spinlock_wait_cycles":40192,"thread_start_us":12843,"threads_started":1}
I20260620 18:32:48.771956 17884 heartbeater.cc:347] Connected to a master server at 127.10.131.61:38067
I20260620 18:32:48.772099 17884 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:48.772596 17884 heartbeater.cc:511] Master 127.10.131.61:38067 requested a full tablet report, sending...
I20260620 18:32:48.774174 17516 ts_manager.cc:194] Registered new tserver with Master: 2694a3ddafbc484e882015fb46f9bbfc (127.10.131.1:45237)
I20260620 18:32:48.855109 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 17112
I20260620 18:32:48.872040 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.2:35827
--local_ip_for_outbound_sockets=127.10.131.2
--tserver_master_addrs=127.10.131.62:44649
--webserver_port=46551
--webserver_interface=127.10.131.2
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.2.keytab
--principal=oryx/127.10.131.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:37493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.131.62:44649,127.10.131.61:38067
--tserver_master_addrs=127.10.131.62:44649,127.10.131.61:38067,127.10.131.60:36193 with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:32:49.072737 17661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2d7e501282a949a1be211a121201e250" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: false } } peers { permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: NON_VOTER last_known_addr { host: "127.10.131.60" port: 36193 } attrs { promote: true } } }
I20260620 18:32:49.073128 17661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:49.074530 17896 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250: attempting to promote NON_VOTER 1d5bf8ef8b624ecfa578fc93e5492446 to VOTER
I20260620 18:32:49.075045 17896 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: false } } peers { permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36193 } attrs { promote: false } }
I20260620 18:32:49.076771 17714 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [term 1 LEARNER]: Refusing update from remote peer 2d7e501282a949a1be211a121201e250: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 18:32:49.076752 17526 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [term 1 FOLLOWER]: Refusing update from remote peer 2d7e501282a949a1be211a121201e250: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 18:32:49.077378 17661 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [LEADER]: Connected to new peer: Peer: permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 18:32:49.077762 17895 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 18:32:49.080001 17661 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1d5bf8ef8b624ecfa578fc93e5492446 (127.10.131.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: false } } peers { permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36193 } attrs { promote: false } } }
I20260620 18:32:49.081933 17526 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1d5bf8ef8b624ecfa578fc93e5492446 (127.10.131.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: false } } peers { permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36193 } attrs { promote: false } } }
I20260620 18:32:49.082727 17714 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1d5bf8ef8b624ecfa578fc93e5492446 (127.10.131.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: false } } peers { permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36193 } attrs { promote: false } } }
I20260620 18:32:49.086344 17661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2d7e501282a949a1be211a121201e250" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: false } } peers { permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36193 } attrs { promote: false } } }
I20260620 18:32:49.086581 17661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:49.086849 17895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2d7e501282a949a1be211a121201e250" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: false } } peers { permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36193 } attrs { promote: false } } }
I20260620 18:32:49.087153 17895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:49.087518 17895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2d7e501282a949a1be211a121201e250" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: false } } peers { permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36193 } attrs { promote: false } } }
I20260620 18:32:49.087764 17895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:49.089293 17900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2d7e501282a949a1be211a121201e250" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: false } } peers { permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36193 } attrs { promote: false } } }
I20260620 18:32:49.089774 17900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:32:49.092123 17899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2d7e501282a949a1be211a121201e250" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: false } } peers { permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36193 } attrs { promote: false } } }
I20260620 18:32:49.092501 17899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 18:32:49.194517 17894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:49.194773 17894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:49.194957 17894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:49.202343 17894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:49.202468 17894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:49.202618 17894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.2
I20260620 18:32:49.212075 17894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:37493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.2.keytab
--principal=oryx/127.10.131.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.2:35827
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.131.2
--webserver_port=46551
--tserver_master_addrs=127.10.131.62:44649,127.10.131.61:38067,127.10.131.60:36193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.2
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:49.213348 17894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:49.214664 17894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:49.227600 17905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:49.227558 17908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:49.227648 17906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:49.229339 17894 server_base.cc:1034] running on GCE node
I20260620 18:32:49.230108 17894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:49.231073 17894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:49.232306 17894 hybrid_clock.cc:648] HybridClock initialized: now 1781980369232259 us; error 52 us; skew 500 ppm
I20260620 18:32:49.232651 17894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:49 dist-test-slave-cbn3 krb5kdc[16872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980369, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:49.236060 17894 init.cc:377] Logged in from keytab as oryx/127.10.131.2@KRBTEST.COM (short username oryx)
I20260620 18:32:49.237663 17894 webserver.cc:460] Webserver started at http://127.10.131.2:46551/ using document root <none> and password file <none>
I20260620 18:32:49.238401 17894 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:49.238571 17894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:49.244199 17894 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 18:32:49.246942 17915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:49.248308 17894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:32:49.248488 17894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-1/wal
uuid: "f6e2a79583ed46ad9d079b7a72b70b30"
format_stamp: "Formatted at 2026-06-20 18:32:46 on dist-test-slave-cbn3"
server_key: "c491c797896a90bc1897f15c163df968"
server_key_iv: "b7be3747a434117b10205a539c1c2143"
server_key_version: "encryptionkey@0"
I20260620 18:32:49.249637 17894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:49.298673 17894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:49.303161 17894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:49.303416 17894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:49.304358 17894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:49.305835 17894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:49.305946 17894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:49.306022 17894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:49.306105 17894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:49.341449 17894 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.2:35827
I20260620 18:32:49.341504 18027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.2:35827 every 8 connection(s)
I20260620 18:32:49.342835 17894 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
I20260620 18:32:49.352207 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 17894
I20260620 18:32:49.352571 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 17245
Jun 20 18:32:49 dist-test-slave-cbn3 krb5kdc[16872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980369, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.2@KRBTEST.COM for oryx/127.10.131.60@KRBTEST.COM
Jun 20 18:32:49 dist-test-slave-cbn3 krb5kdc[16872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.131.3@KRBTEST.COM
Jun 20 18:32:49 dist-test-slave-cbn3 krb5kdc[16872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980369, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.2@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM
Jun 20 18:32:49 dist-test-slave-cbn3 krb5kdc[16872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980369, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.2@KRBTEST.COM for oryx/127.10.131.61@KRBTEST.COM
I20260620 18:32:49.396147 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:49.345905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:52925 (local address 127.10.131.60:36193)
0620 18:32:49.373733 (+ 27828us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:49.373747 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:49.373772 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:32:49.373841 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:49.373846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:49.373924 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:49.374478 (+ 554us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:49.376987 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:49.378224 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:49.382202 (+ 3978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:49.382859 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:49.383033 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:49.389045 (+ 6012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:49.389053 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:49.389061 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:49.389112 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:49.392434 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:49.393390 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:49.393396 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:49.393401 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:49.393538 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:49.394051 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:49.394057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:49.394063 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:49.394413 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:49.394540 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:49.395530 (+ 990us) server_negotiation.cc:299] Negotiation successful
0620 18:32:49.395760 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27679,"thread_start_us":27569,"threads_started":1}
I20260620 18:32:49.396173 18039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:49.353716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52202 (local address 127.10.131.3:34005)
0620 18:32:49.374055 (+ 20339us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:49.374067 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:49.374100 (+ 33us) server_negotiation.cc:407] Connection header received
0620 18:32:49.374168 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:49.374176 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:49.374427 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:49.374574 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:49.376946 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:49.378520 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:49.380558 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:49.381490 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:49.381602 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:49.388240 (+ 6638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:49.388259 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:49.388279 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:49.388324 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:49.392647 (+ 4323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:49.393680 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:49.393692 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:49.393696 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:49.393853 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:49.394300 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:49.394305 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:49.394311 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:49.394736 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:49.395011 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:49.395494 (+ 483us) server_negotiation.cc:299] Negotiation successful
0620 18:32:49.395754 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20167,"thread_start_us":19989,"threads_started":1}
I20260620 18:32:49.397527 17368 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:49.397845 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:49.346761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:36815 (local address 127.10.131.61:38067)
0620 18:32:49.373219 (+ 26458us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:49.373226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:49.379555 (+ 6329us) server_negotiation.cc:407] Connection header received
0620 18:32:49.379672 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:49.379677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:49.379750 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:49.379859 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:49.381628 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:49.382766 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:49.384899 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:49.385505 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:49.385612 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:49.390755 (+ 5143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:49.390761 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:49.390787 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:49.390836 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:49.394374 (+ 3538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:49.395202 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:49.395208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:49.395211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:49.395344 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:49.395737 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:49.395742 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:49.395745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:49.396065 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:49.396204 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:49.397335 (+ 1131us) server_negotiation.cc:299] Negotiation successful
0620 18:32:49.397546 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26342,"thread_start_us":26458,"threads_started":1}
I20260620 18:32:49.398281 18037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:49.346454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:36193 (local address 127.10.131.2:52925)
0620 18:32:49.368174 (+ 21720us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:49.368243 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:49.371486 (+ 3243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:49.374980 (+ 3494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:49.374996 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:49.375840 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:49.376765 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:49.376796 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:49.378919 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:49.378928 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:49.381991 (+ 3063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:49.382005 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:49.383355 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:49.383380 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:49.383680 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:49.384393 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:49.384422 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:49.387866 (+ 3444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:49.392600 (+ 4734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:49.392612 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:49.392634 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:49.393239 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:49.393686 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:49.393690 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:49.393694 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:49.393925 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:49.394633 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:49.394640 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:49.395088 (+ 448us) client_negotiation.cc:769] Sending connection context
0620 18:32:49.396348 (+ 1260us) client_negotiation.cc:240] Negotiation successful
0620 18:32:49.396911 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21299,"thread_start_us":31897,"threads_started":1}
I20260620 18:32:49.398739 18035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:49.346487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:38067 (local address 127.10.131.2:36815)
0620 18:32:49.379452 (+ 32965us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:49.379466 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:49.379571 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:49.379928 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:49.379933 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:49.381066 (+ 1133us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:49.381443 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:49.381454 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:49.383000 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:49.383007 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:49.384681 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:49.384694 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:49.385947 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:49.385955 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:49.386189 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:49.388123 (+ 1934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:49.388147 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:49.390541 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:49.394521 (+ 3980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:49.394526 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:49.394530 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:49.395067 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:49.395485 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:49.395490 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:49.395494 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:49.395618 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:49.396285 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:49.396289 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:49.396433 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 18:32:49.398027 (+ 1594us) client_negotiation.cc:240] Negotiation successful
0620 18:32:49.398318 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32855,"mutex_wait_us":485,"thread_start_us":25072,"threads_started":1}
I20260620 18:32:49.399785 18028 heartbeater.cc:347] Connected to a master server at 127.10.131.60:36193
I20260620 18:32:49.400128 18028 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:49.399816 18030 heartbeater.cc:347] Connected to a master server at 127.10.131.61:38067
I20260620 18:32:49.401021 18030 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:49.401041 18028 heartbeater.cc:511] Master 127.10.131.60:36193 requested a full tablet report, sending...
I20260620 18:32:49.401465 18030 heartbeater.cc:511] Master 127.10.131.61:38067 requested a full tablet report, sending...
I20260620 18:32:49.402837 17704 ts_manager.cc:194] Registered new tserver with Master: f6e2a79583ed46ad9d079b7a72b70b30 (127.10.131.2:35827)
I20260620 18:32:49.402936 17516 ts_manager.cc:194] Registered new tserver with Master: f6e2a79583ed46ad9d079b7a72b70b30 (127.10.131.2:35827)
I20260620 18:32:49.405260 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:49.346454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.10.131.2:51659)
0620 18:32:49.378789 (+ 32335us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:49.378812 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:49.378994 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:49.380073 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:49.380078 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:49.380883 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:49.381251 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:49.381262 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:49.382723 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:49.382730 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:49.384623 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:49.384643 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:49.385988 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:49.385998 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:49.386225 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:49.386959 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:49.386977 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:49.390291 (+ 3314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:49.396597 (+ 6306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:49.396602 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:49.396605 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:49.397092 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:49.403574 (+ 6482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:49.403579 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:49.403582 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:49.403756 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:49.404277 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:49.404282 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:49.404425 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 18:32:49.404554 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 18:32:49.404824 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32162,"mutex_wait_us":36,"spinlock_wait_cycles":31616,"thread_start_us":32548,"threads_started":1}
I20260620 18:32:49.405300 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:49.346235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:51659 (local address 127.10.131.62:44649)
0620 18:32:49.374322 (+ 28087us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:49.374328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:49.378891 (+ 4563us) server_negotiation.cc:407] Connection header received
0620 18:32:49.379212 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:49.379218 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:49.379308 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:49.379400 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:49.381467 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:49.382526 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:49.384799 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:49.385587 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:49.385700 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:49.390506 (+ 4806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:49.390514 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:49.390521 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:49.390577 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:49.394830 (+ 4253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:49.402396 (+ 7566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:49.402402 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:49.402406 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:49.402561 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:49.403859 (+ 1298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:49.403865 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:49.403868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:49.404156 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:49.404311 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:49.404655 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 18:32:49.404925 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27976,"thread_start_us":34178,"threads_started":1}
I20260620 18:32:49.406347 18029 heartbeater.cc:347] Connected to a master server at 127.10.131.62:44649
I20260620 18:32:49.406479 18029 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:49.406832 18029 heartbeater.cc:511] Master 127.10.131.62:44649 requested a full tablet report, sending...
I20260620 18:32:49.407794 16917 ts_manager.cc:194] Re-registered known tserver with Master: f6e2a79583ed46ad9d079b7a72b70b30 (127.10.131.2:35827)
I20260620 18:32:49.408579 16917 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.131.2@KRBTEST.COM'} at 127.10.131.2:51659
I20260620 18:32:49.483537 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 17245
I20260620 18:32:49.499997 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.3:34005
--local_ip_for_outbound_sockets=127.10.131.3
--tserver_master_addrs=127.10.131.62:44649
--webserver_port=44981
--webserver_interface=127.10.131.3
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.3.keytab
--principal=oryx/127.10.131.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:37493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.131.62:44649,127.10.131.61:38067
--tserver_master_addrs=127.10.131.62:44649,127.10.131.61:38067,127.10.131.60:36193 with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:32:49.771504 17883 heartbeater.cc:503] Master 127.10.131.62:44649 was elected leader, sending a full tablet report...
W20260620 18:32:49.825505 18041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:49.825771 18041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:49.825865 18041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:49.833246 18041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:49.833371 18041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:49.833498 18041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.3
I20260620 18:32:49.842957 18041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:37493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.3.keytab
--principal=oryx/127.10.131.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.3:34005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.131.3
--webserver_port=44981
--tserver_master_addrs=127.10.131.62:44649,127.10.131.61:38067,127.10.131.60:36193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.3
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:49.844177 18041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:49.845445 18041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:49.858445 18046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:49.858611 18047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:49.858649 18049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:49.859961 18041 server_base.cc:1034] running on GCE node
I20260620 18:32:49.860558 18041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:49.861588 18041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:49.862779 18041 hybrid_clock.cc:648] HybridClock initialized: now 1781980369862726 us; error 46 us; skew 500 ppm
I20260620 18:32:49.863096 18041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:49 dist-test-slave-cbn3 krb5kdc[16872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980369, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:49.866251 18041 init.cc:377] Logged in from keytab as oryx/127.10.131.3@KRBTEST.COM (short username oryx)
I20260620 18:32:49.867677 18041 webserver.cc:460] Webserver started at http://127.10.131.3:44981/ using document root <none> and password file <none>
I20260620 18:32:49.868355 18041 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:49.868494 18041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:49.873607 18041 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260620 18:32:49.876052 18056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:49.877307 18041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:32:49.877457 18041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-2/wal
uuid: "4e122b7d3c6f4e4b934949448ccd2fa5"
format_stamp: "Formatted at 2026-06-20 18:32:46 on dist-test-slave-cbn3"
server_key: "13cf5633ce4997965f8f4d21a811bab6"
server_key_iv: "cce1f7f207eeaeecdb1e697ee63c77d4"
server_key_version: "encryptionkey@0"
I20260620 18:32:49.878386 18041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:49.918138 18041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:49.922353 18041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:49.922652 18041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:49.923619 18041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:32:49.925041 18041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:49.925132 18041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:49.925269 18041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:49.925345 18041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:49.957890 18041 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.3:34005
I20260620 18:32:49.957976 18168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.3:34005 every 8 connection(s)
I20260620 18:32:49.959322 18041 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
I20260620 18:32:49.969583 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 18041
Jun 20 18:32:49 dist-test-slave-cbn3 krb5kdc[16872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980369, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.3@KRBTEST.COM for oryx/127.10.131.60@KRBTEST.COM
Jun 20 18:32:49 dist-test-slave-cbn3 krb5kdc[16872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980369, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.3@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM
I20260620 18:32:50.009883 18175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:49.962519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:59269 (local address 127.10.131.60:36193)
0620 18:32:49.981279 (+ 18760us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:49.981288 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:49.989608 (+ 8320us) server_negotiation.cc:407] Connection header received
0620 18:32:49.989901 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:49.989910 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:49.990015 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:49.990121 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:49.991534 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:49.992683 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:49.994597 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:49.995200 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:49.995293 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.000910 (+ 5617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:50.000917 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:50.000922 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:50.000961 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:50.005010 (+ 4049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:50.006227 (+ 1217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:50.006236 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:50.006242 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:50.006414 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:50.007049 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:50.007055 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:50.007061 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:50.007523 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:50.007672 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:50.009125 (+ 1453us) server_negotiation.cc:299] Negotiation successful
0620 18:32:50.009454 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18591,"thread_start_us":18496,"threads_started":1}
I20260620 18:32:50.011284 18174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:49.962508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:53285 (local address 127.10.131.62:44649)
0620 18:32:49.977885 (+ 15377us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:49.977894 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:49.985645 (+ 7751us) server_negotiation.cc:407] Connection header received
0620 18:32:49.987672 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:49.987678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:49.987769 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:49.987882 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:49.989978 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:49.991100 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:49.994360 (+ 3260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:49.995371 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:49.995506 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.001644 (+ 6138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:50.001653 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:50.001661 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:50.001709 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:50.006033 (+ 4324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:50.007840 (+ 1807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:50.007848 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:50.007855 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:50.008060 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:50.008745 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:50.008750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:50.008753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:50.009269 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:50.009446 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:50.010452 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0620 18:32:50.010765 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15210,"thread_start_us":15409,"threads_started":1}
I20260620 18:32:50.012159 18178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:49.962925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:36193 (local address 127.10.131.3:59269)
0620 18:32:49.989482 (+ 26557us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:49.989507 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:49.989722 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:49.990313 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:49.990317 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:49.990951 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:49.991285 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:49.991299 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:49.992906 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:49.992912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:49.994435 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:49.994446 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:49.995622 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:49.995631 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:49.995983 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:49.996837 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:49.996892 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:50.000356 (+ 3464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:50.005205 (+ 4849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:50.005218 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:50.005242 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:50.006064 (+ 822us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:50.006596 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:50.006604 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:50.006610 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:50.006898 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:50.007773 (+ 875us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:50.007785 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:50.008460 (+ 675us) client_negotiation.cc:769] Sending connection context
0620 18:32:50.010268 (+ 1808us) client_negotiation.cc:240] Negotiation successful
0620 18:32:50.010524 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26347,"mutex_wait_us":22,"thread_start_us":26561,"threads_started":1}
Jun 20 18:32:50 dist-test-slave-cbn3 krb5kdc[16872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980369, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.3@KRBTEST.COM for oryx/127.10.131.61@KRBTEST.COM
I20260620 18:32:50.013164 18177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:49.962785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.10.131.3:53285)
0620 18:32:49.985460 (+ 22675us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:49.985510 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:49.987417 (+ 1907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:49.988124 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:49.988134 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:49.988653 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:49.989766 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:49.989794 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:49.992109 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:49.992114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:49.994208 (+ 2094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:49.994227 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:49.995706 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:49.995712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:49.997776 (+ 2064us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:49.999014 (+ 1238us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:49.999030 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:50.001438 (+ 2408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:50.007262 (+ 5824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:50.007267 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:50.007270 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:50.007731 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:50.008450 (+ 719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:50.008454 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:50.008457 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:50.008623 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:50.009411 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:50.009416 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:50.009564 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 18:32:50.009721 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 18:32:50.010349 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22399,"mutex_wait_us":1857,"thread_start_us":26121,"threads_started":1}
I20260620 18:32:50.014396 18169 heartbeater.cc:347] Connected to a master server at 127.10.131.60:36193
I20260620 18:32:50.014555 18170 heartbeater.cc:347] Connected to a master server at 127.10.131.62:44649
I20260620 18:32:50.014806 18170 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:50.014812 18169 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:50.015879 18170 heartbeater.cc:511] Master 127.10.131.62:44649 requested a full tablet report, sending...
I20260620 18:32:50.017498 18169 heartbeater.cc:511] Master 127.10.131.60:36193 requested a full tablet report, sending...
I20260620 18:32:50.018054 16917 ts_manager.cc:194] Re-registered known tserver with Master: 4e122b7d3c6f4e4b934949448ccd2fa5 (127.10.131.3:34005)
I20260620 18:32:50.018826 17704 ts_manager.cc:194] Registered new tserver with Master: 4e122b7d3c6f4e4b934949448ccd2fa5 (127.10.131.3:34005)
I20260620 18:32:50.019498 16917 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.131.3@KRBTEST.COM'} at 127.10.131.3:53285
I20260620 18:32:50.023061 18173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:49.962212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:57995 (local address 127.10.131.61:38067)
0620 18:32:49.981631 (+ 19419us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:49.981635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:49.990254 (+ 8619us) server_negotiation.cc:407] Connection header received
0620 18:32:49.990485 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:49.990492 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:49.990584 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:49.990673 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:49.994410 (+ 3737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:49.995517 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:49.998102 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:49.998741 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:49.998813 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.014702 (+ 15889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:50.014713 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:50.014718 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:50.014773 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:50.019009 (+ 4236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:50.019885 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:50.019896 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:50.019902 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:50.020126 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:50.020733 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:50.020741 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:50.020747 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:50.021225 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:50.021421 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:50.022129 (+ 708us) server_negotiation.cc:299] Negotiation successful
0620 18:32:50.022449 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19312,"thread_start_us":19190,"threads_started":1}
I20260620 18:32:50.024653 18176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:49.962785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:38067 (local address 127.10.131.3:57995)
0620 18:32:49.990163 (+ 27378us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:49.990181 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:49.990336 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:49.990925 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:49.990930 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:49.991657 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:49.991935 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:49.991946 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:49.995978 (+ 4032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:49.995983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:49.997636 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:49.997651 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.000208 (+ 2557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.000214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:50.000389 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.001661 (+ 1272us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:50.001690 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:50.014277 (+ 12587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:50.019184 (+ 4907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:50.019190 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:50.019194 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:50.019712 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:50.020313 (+ 601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:50.020322 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:50.020328 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:50.020585 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:50.021546 (+ 961us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:50.021554 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:50.021776 (+ 222us) client_negotiation.cc:769] Sending connection context
0620 18:32:50.023534 (+ 1758us) client_negotiation.cc:240] Negotiation successful
0620 18:32:50.024007 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27251,"mutex_wait_us":151,"spinlock_wait_cycles":35584,"thread_start_us":22181,"threads_started":1}
I20260620 18:32:50.026152 18171 heartbeater.cc:347] Connected to a master server at 127.10.131.61:38067
I20260620 18:32:50.026317 18171 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:50.026803 18171 heartbeater.cc:511] Master 127.10.131.61:38067 requested a full tablet report, sending...
I20260620 18:32:50.028259 17516 ts_manager.cc:194] Registered new tserver with Master: 4e122b7d3c6f4e4b934949448ccd2fa5 (127.10.131.3:34005)
I20260620 18:32:50.041721 18174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:49.991322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46190 (local address 127.10.131.62:44649)
0620 18:32:50.011578 (+ 20256us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:50.011583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:50.012161 (+ 578us) server_negotiation.cc:407] Connection header received
0620 18:32:50.013101 (+ 940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:50.013110 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:50.013217 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:50.013290 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:50.020878 (+ 7588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.021935 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.024616 (+ 2681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.025806 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.025926 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.032508 (+ 6582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:50.032518 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:50.032525 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:50.032580 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:50.038510 (+ 5930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:50.039408 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:50.039415 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:50.039419 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:50.039581 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:50.040005 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:50.040010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:50.040015 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:50.040373 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:50.040548 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:50.041070 (+ 522us) server_negotiation.cc:299] Negotiation successful
0620 18:32:50.041318 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20169,"thread_start_us":25447,"threads_started":1}
I20260620 18:32:50.051383 18173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:49.995782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39280 (local address 127.10.131.61:38067)
0620 18:32:50.023470 (+ 27688us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:50.023478 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:50.030271 (+ 6793us) server_negotiation.cc:407] Connection header received
0620 18:32:50.030496 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:50.030513 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:50.030630 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:50.030745 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:50.037080 (+ 6335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.038293 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.040238 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.040980 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.041095 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.045520 (+ 4425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:50.045525 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:50.045528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:50.045599 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:50.048382 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:50.049245 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:50.049254 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:50.049259 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:50.049407 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:50.049809 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:50.049816 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:50.049822 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:50.050158 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:50.050438 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:50.050788 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 18:32:50.051010 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27544,"thread_start_us":38087,"threads_started":1}
I20260620 18:32:50.054450 18175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.003169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50898 (local address 127.10.131.60:36193)
0620 18:32:50.010224 (+ 7055us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:50.010232 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:50.035051 (+ 24819us) server_negotiation.cc:407] Connection header received
0620 18:32:50.035275 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:50.035284 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:50.035396 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:50.035503 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:50.037954 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.038799 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.041739 (+ 2940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.042365 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.042442 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.045675 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:50.045682 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:50.045690 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:50.045731 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:50.051515 (+ 5784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:50.052285 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:50.052291 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:50.052294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:50.052457 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:50.052928 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:50.052933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:50.052937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:50.053266 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:50.053529 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:50.053843 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 18:32:50.054066 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6953,"thread_start_us":31567,"threads_started":1}
I20260620 18:32:50.047259 16917 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:32:50.056742 16917 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:32:50.091404 18200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.076160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38298 (local address 127.10.131.2:35827)
0620 18:32:50.082015 (+ 5855us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:50.082028 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:50.082069 (+ 41us) server_negotiation.cc:407] Connection header received
0620 18:32:50.082200 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:50.082212 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:50.082599 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:50.084194 (+ 1595us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:50.085086 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.086488 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.089359 (+ 2871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.090360 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.090518 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.090747 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:50.090950 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 18:32:50.091117 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5681,"thread_start_us":5590,"threads_started":1}
I20260620 18:32:50.091507 18197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.075882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:35827 (local address 127.0.0.1:38298)
0620 18:32:50.077249 (+ 1367us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:50.077262 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:50.077377 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:50.084177 (+ 6800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:50.084181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:50.084208 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:50.084734 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:50.084748 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.086652 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.086660 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:50.089190 (+ 2530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:50.089204 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.090565 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.090571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:50.090744 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.090787 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 18:32:50.090891 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 18:32:50.091034 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1278,"mutex_wait_us":160,"thread_start_us":405,"threads_started":1}
I20260620 18:32:50.092792 18198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.075685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52204 (local address 127.10.131.3:34005)
0620 18:32:50.083292 (+ 7607us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:50.083297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:50.083324 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:32:50.083445 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:50.083453 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:50.083693 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:50.083882 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:50.084671 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.086452 (+ 1781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.089602 (+ 3150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.091149 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.091270 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.091655 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:50.092295 (+ 640us) server_negotiation.cc:299] Negotiation successful
0620 18:32:50.092484 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7452,"thread_start_us":7297,"threads_started":1}
I20260620 18:32:50.092835 18199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.075455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:34005 (local address 127.0.0.1:52204)
0620 18:32:50.076553 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:50.076570 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:50.076666 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:50.083904 (+ 7238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:50.083912 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:50.083949 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:50.084476 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:50.084492 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.087049 (+ 2557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.087054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:50.089470 (+ 2416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:50.089479 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.091866 (+ 2387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.091891 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:50.092102 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.092131 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 18:32:50.092232 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 18:32:50.092521 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":968,"mutex_wait_us":8,"thread_start_us":1596,"threads_started":1}
I20260620 18:32:50.095710 18202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.077714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50564 (local address 127.10.131.1:45237)
0620 18:32:50.086956 (+ 9242us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:50.086964 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:50.086991 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:32:50.087090 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:50.087098 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:50.087314 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:50.087515 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:50.088382 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.089512 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.093420 (+ 3908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.094622 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.094781 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.094983 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:50.095223 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 18:32:50.095390 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9064,"thread_start_us":8937,"threads_started":1}
I20260620 18:32:50.095877 18201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.077431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:45237 (local address 127.0.0.1:50564)
0620 18:32:50.078181 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:50.078215 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:50.078344 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:50.087731 (+ 9387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:50.087736 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:50.087772 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:50.088184 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:50.088192 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.089635 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.089640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:50.093272 (+ 3632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:50.093285 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.094831 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.094837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:50.095027 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.095059 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 18:32:50.095162 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 18:32:50.095542 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":344,"threads_started":1}
I20260620 18:32:50.096009 17963 tablet_service.cc:1467] Processing CreateTablet for tablet 10e88367b7064efc9dc995574ba0a1d8 (DEFAULT_TABLE table=test-table [id=c28b93e196104fa4bf8287584b8df4fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:50.096339 18104 tablet_service.cc:1467] Processing CreateTablet for tablet 10e88367b7064efc9dc995574ba0a1d8 (DEFAULT_TABLE table=test-table [id=c28b93e196104fa4bf8287584b8df4fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:50.097561 18104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e88367b7064efc9dc995574ba0a1d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:50.098440 17963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e88367b7064efc9dc995574ba0a1d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:50.098976 17817 tablet_service.cc:1467] Processing CreateTablet for tablet 10e88367b7064efc9dc995574ba0a1d8 (DEFAULT_TABLE table=test-table [id=c28b93e196104fa4bf8287584b8df4fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:50.099974 17817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e88367b7064efc9dc995574ba0a1d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:50.110721 18203 tablet_bootstrap.cc:492] T 10e88367b7064efc9dc995574ba0a1d8 P 2694a3ddafbc484e882015fb46f9bbfc: Bootstrap starting.
I20260620 18:32:50.112578 18204 tablet_bootstrap.cc:492] T 10e88367b7064efc9dc995574ba0a1d8 P 4e122b7d3c6f4e4b934949448ccd2fa5: Bootstrap starting.
I20260620 18:32:50.113354 18205 tablet_bootstrap.cc:492] T 10e88367b7064efc9dc995574ba0a1d8 P f6e2a79583ed46ad9d079b7a72b70b30: Bootstrap starting.
I20260620 18:32:50.115338 18203 tablet_bootstrap.cc:654] T 10e88367b7064efc9dc995574ba0a1d8 P 2694a3ddafbc484e882015fb46f9bbfc: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:50.116128 18205 tablet_bootstrap.cc:654] T 10e88367b7064efc9dc995574ba0a1d8 P f6e2a79583ed46ad9d079b7a72b70b30: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:50.116432 18204 tablet_bootstrap.cc:654] T 10e88367b7064efc9dc995574ba0a1d8 P 4e122b7d3c6f4e4b934949448ccd2fa5: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:50.116588 18203 log.cc:824] T 10e88367b7064efc9dc995574ba0a1d8 P 2694a3ddafbc484e882015fb46f9bbfc: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:50.117238 18205 log.cc:824] T 10e88367b7064efc9dc995574ba0a1d8 P f6e2a79583ed46ad9d079b7a72b70b30: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:50.117712 18204 log.cc:824] T 10e88367b7064efc9dc995574ba0a1d8 P 4e122b7d3c6f4e4b934949448ccd2fa5: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:50.119655 18203 tablet_bootstrap.cc:492] T 10e88367b7064efc9dc995574ba0a1d8 P 2694a3ddafbc484e882015fb46f9bbfc: No bootstrap required, opened a new log
I20260620 18:32:50.119943 18203 ts_tablet_manager.cc:1397] T 10e88367b7064efc9dc995574ba0a1d8 P 2694a3ddafbc484e882015fb46f9bbfc: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 18:32:50.120484 18204 tablet_bootstrap.cc:492] T 10e88367b7064efc9dc995574ba0a1d8 P 4e122b7d3c6f4e4b934949448ccd2fa5: No bootstrap required, opened a new log
I20260620 18:32:50.120754 18204 ts_tablet_manager.cc:1397] T 10e88367b7064efc9dc995574ba0a1d8 P 4e122b7d3c6f4e4b934949448ccd2fa5: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 18:32:50.121608 18205 tablet_bootstrap.cc:492] T 10e88367b7064efc9dc995574ba0a1d8 P f6e2a79583ed46ad9d079b7a72b70b30: No bootstrap required, opened a new log
I20260620 18:32:50.121841 18205 ts_tablet_manager.cc:1397] T 10e88367b7064efc9dc995574ba0a1d8 P f6e2a79583ed46ad9d079b7a72b70b30: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.002s
I20260620 18:32:50.129184 18205 raft_consensus.cc:348] T 10e88367b7064efc9dc995574ba0a1d8 P f6e2a79583ed46ad9d079b7a72b70b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e122b7d3c6f4e4b934949448ccd2fa5" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 34005 } } peers { permanent_uuid: "f6e2a79583ed46ad9d079b7a72b70b30" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 35827 } } peers { permanent_uuid: "2694a3ddafbc484e882015fb46f9bbfc" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 45237 } }
I20260620 18:32:50.129595 18205 raft_consensus.cc:374] T 10e88367b7064efc9dc995574ba0a1d8 P f6e2a79583ed46ad9d079b7a72b70b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:50.129694 18205 raft_consensus.cc:729] T 10e88367b7064efc9dc995574ba0a1d8 P f6e2a79583ed46ad9d079b7a72b70b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6e2a79583ed46ad9d079b7a72b70b30, State: Initialized, Role: FOLLOWER
I20260620 18:32:50.130283 18205 consensus_queue.cc:260] T 10e88367b7064efc9dc995574ba0a1d8 P f6e2a79583ed46ad9d079b7a72b70b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e122b7d3c6f4e4b934949448ccd2fa5" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 34005 } } peers { permanent_uuid: "f6e2a79583ed46ad9d079b7a72b70b30" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 35827 } } peers { permanent_uuid: "2694a3ddafbc484e882015fb46f9bbfc" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 45237 } }
I20260620 18:32:50.130789 18203 raft_consensus.cc:348] T 10e88367b7064efc9dc995574ba0a1d8 P 2694a3ddafbc484e882015fb46f9bbfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e122b7d3c6f4e4b934949448ccd2fa5" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 34005 } } peers { permanent_uuid: "f6e2a79583ed46ad9d079b7a72b70b30" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 35827 } } peers { permanent_uuid: "2694a3ddafbc484e882015fb46f9bbfc" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 45237 } }
I20260620 18:32:50.131242 18203 raft_consensus.cc:374] T 10e88367b7064efc9dc995574ba0a1d8 P 2694a3ddafbc484e882015fb46f9bbfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:50.131342 18203 raft_consensus.cc:729] T 10e88367b7064efc9dc995574ba0a1d8 P 2694a3ddafbc484e882015fb46f9bbfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2694a3ddafbc484e882015fb46f9bbfc, State: Initialized, Role: FOLLOWER
I20260620 18:32:50.131255 18204 raft_consensus.cc:348] T 10e88367b7064efc9dc995574ba0a1d8 P 4e122b7d3c6f4e4b934949448ccd2fa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e122b7d3c6f4e4b934949448ccd2fa5" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 34005 } } peers { permanent_uuid: "f6e2a79583ed46ad9d079b7a72b70b30" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 35827 } } peers { permanent_uuid: "2694a3ddafbc484e882015fb46f9bbfc" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 45237 } }
I20260620 18:32:50.131654 18204 raft_consensus.cc:374] T 10e88367b7064efc9dc995574ba0a1d8 P 4e122b7d3c6f4e4b934949448ccd2fa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:50.131978 18205 ts_tablet_manager.cc:1428] T 10e88367b7064efc9dc995574ba0a1d8 P f6e2a79583ed46ad9d079b7a72b70b30: Time spent starting tablet: real 0.010s user 0.009s sys 0.001s
I20260620 18:32:50.132061 18204 raft_consensus.cc:729] T 10e88367b7064efc9dc995574ba0a1d8 P 4e122b7d3c6f4e4b934949448ccd2fa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e122b7d3c6f4e4b934949448ccd2fa5, State: Initialized, Role: FOLLOWER
I20260620 18:32:50.131942 18203 consensus_queue.cc:260] T 10e88367b7064efc9dc995574ba0a1d8 P 2694a3ddafbc484e882015fb46f9bbfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e122b7d3c6f4e4b934949448ccd2fa5" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 34005 } } peers { permanent_uuid: "f6e2a79583ed46ad9d079b7a72b70b30" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 35827 } } peers { permanent_uuid: "2694a3ddafbc484e882015fb46f9bbfc" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 45237 } }
I20260620 18:32:50.132220 18029 heartbeater.cc:503] Master 127.10.131.62:44649 was elected leader, sending a full tablet report...
I20260620 18:32:50.132723 18204 consensus_queue.cc:260] T 10e88367b7064efc9dc995574ba0a1d8 P 4e122b7d3c6f4e4b934949448ccd2fa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e122b7d3c6f4e4b934949448ccd2fa5" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 34005 } } peers { permanent_uuid: "f6e2a79583ed46ad9d079b7a72b70b30" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 35827 } } peers { permanent_uuid: "2694a3ddafbc484e882015fb46f9bbfc" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 45237 } }
I20260620 18:32:50.143656 18203 ts_tablet_manager.cc:1428] T 10e88367b7064efc9dc995574ba0a1d8 P 2694a3ddafbc484e882015fb46f9bbfc: Time spent starting tablet: real 0.024s user 0.007s sys 0.012s
I20260620 18:32:50.145269 18204 ts_tablet_manager.cc:1428] T 10e88367b7064efc9dc995574ba0a1d8 P 4e122b7d3c6f4e4b934949448ccd2fa5: Time spent starting tablet: real 0.024s user 0.016s sys 0.009s
I20260620 18:32:50.145532 18170 heartbeater.cc:503] Master 127.10.131.62:44649 was elected leader, sending a full tablet report...
W20260620 18:32:50.225106 17885 tablet.cc:2369] T 10e88367b7064efc9dc995574ba0a1d8 P 2694a3ddafbc484e882015fb46f9bbfc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:32:50.229076 18172 tablet.cc:2369] T 10e88367b7064efc9dc995574ba0a1d8 P 4e122b7d3c6f4e4b934949448ccd2fa5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:32:50.311550 18210 raft_consensus.cc:482] T 10e88367b7064efc9dc995574ba0a1d8 P 2694a3ddafbc484e882015fb46f9bbfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:32:50.312000 18210 raft_consensus.cc:504] T 10e88367b7064efc9dc995574ba0a1d8 P 2694a3ddafbc484e882015fb46f9bbfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e122b7d3c6f4e4b934949448ccd2fa5" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 34005 } } peers { permanent_uuid: "f6e2a79583ed46ad9d079b7a72b70b30" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 35827 } } peers { permanent_uuid: "2694a3ddafbc484e882015fb46f9bbfc" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 45237 } }
I20260620 18:32:50.313850 18210 leader_election.cc:288] T 10e88367b7064efc9dc995574ba0a1d8 P 2694a3ddafbc484e882015fb46f9bbfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e122b7d3c6f4e4b934949448ccd2fa5 (127.10.131.3:34005), f6e2a79583ed46ad9d079b7a72b70b30 (127.10.131.2:35827)
I20260620 18:32:50.320132 18198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.314120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:36827 (local address 127.10.131.3:34005)
0620 18:32:50.314344 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:50.314353 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:50.315029 (+ 676us) server_negotiation.cc:407] Connection header received
0620 18:32:50.315261 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:50.315265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:50.315324 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:50.315423 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:50.315935 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.316645 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.318451 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.319182 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.319377 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.319452 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:50.319750 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 18:32:50.319885 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:32:50.320477 18200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.314368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:52257 (local address 127.10.131.2:35827)
0620 18:32:50.314631 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:50.314640 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:50.315266 (+ 626us) server_negotiation.cc:407] Connection header received
0620 18:32:50.315463 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:50.315467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:50.315525 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:50.315624 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:50.316168 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.316890 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.318621 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.319526 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.319683 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.319751 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:50.320105 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 18:32:50.320236 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 18:32:50.320695 18213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.314144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:35827 (local address 127.10.131.1:52257)
0620 18:32:50.315156 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:50.315170 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:50.315318 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:50.315646 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:50.315650 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:50.315677 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:50.316014 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:50.316022 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.317041 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.317053 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:50.318458 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:50.318469 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.319786 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.319792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:50.319939 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.319970 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 18:32:50.320058 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 18:32:50.320287 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"thread_start_us":708,"threads_started":1}
I20260620 18:32:50.320815 18212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.314046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:34005 (local address 127.10.131.1:36827)
0620 18:32:50.314907 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:50.314926 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:50.315109 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:50.315438 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:50.315443 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:50.315465 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:50.315788 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:50.315800 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.316783 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.316789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:50.318295 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:50.318307 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.319390 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.319396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:50.319547 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.319570 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 18:32:50.320364 (+ 794us) client_negotiation.cc:240] Negotiation successful
0620 18:32:50.320492 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"thread_start_us":490,"threads_started":1}
I20260620 18:32:50.321880 17983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e88367b7064efc9dc995574ba0a1d8" candidate_uuid: "2694a3ddafbc484e882015fb46f9bbfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6e2a79583ed46ad9d079b7a72b70b30" is_pre_election: true
I20260620 18:32:50.321877 18124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e88367b7064efc9dc995574ba0a1d8" candidate_uuid: "2694a3ddafbc484e882015fb46f9bbfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e122b7d3c6f4e4b934949448ccd2fa5" is_pre_election: true
I20260620 18:32:50.322440 18124 raft_consensus.cc:2454] T 10e88367b7064efc9dc995574ba0a1d8 P 4e122b7d3c6f4e4b934949448ccd2fa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2694a3ddafbc484e882015fb46f9bbfc in term 0.
I20260620 18:32:50.322440 17983 raft_consensus.cc:2454] T 10e88367b7064efc9dc995574ba0a1d8 P f6e2a79583ed46ad9d079b7a72b70b30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2694a3ddafbc484e882015fb46f9bbfc in term 0.
I20260620 18:32:50.323194 17771 leader_election.cc:302] T 10e88367b7064efc9dc995574ba0a1d8 P 2694a3ddafbc484e882015fb46f9bbfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2694a3ddafbc484e882015fb46f9bbfc, 4e122b7d3c6f4e4b934949448ccd2fa5, f6e2a79583ed46ad9d079b7a72b70b30; no voters:
I20260620 18:32:50.323640 18210 raft_consensus.cc:2781] T 10e88367b7064efc9dc995574ba0a1d8 P 2694a3ddafbc484e882015fb46f9bbfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:32:50.323810 18210 raft_consensus.cc:482] T 10e88367b7064efc9dc995574ba0a1d8 P 2694a3ddafbc484e882015fb46f9bbfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:32:50.323908 18210 raft_consensus.cc:3037] T 10e88367b7064efc9dc995574ba0a1d8 P 2694a3ddafbc484e882015fb46f9bbfc [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:50.326937 18210 raft_consensus.cc:504] T 10e88367b7064efc9dc995574ba0a1d8 P 2694a3ddafbc484e882015fb46f9bbfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e122b7d3c6f4e4b934949448ccd2fa5" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 34005 } } peers { permanent_uuid: "f6e2a79583ed46ad9d079b7a72b70b30" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 35827 } } peers { permanent_uuid: "2694a3ddafbc484e882015fb46f9bbfc" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 45237 } }
I20260620 18:32:50.327564 18210 leader_election.cc:288] T 10e88367b7064efc9dc995574ba0a1d8 P 2694a3ddafbc484e882015fb46f9bbfc [CANDIDATE]: Term 1 election: Requested vote from peers 4e122b7d3c6f4e4b934949448ccd2fa5 (127.10.131.3:34005), f6e2a79583ed46ad9d079b7a72b70b30 (127.10.131.2:35827)
I20260620 18:32:50.327978 18124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e88367b7064efc9dc995574ba0a1d8" candidate_uuid: "2694a3ddafbc484e882015fb46f9bbfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e122b7d3c6f4e4b934949448ccd2fa5"
I20260620 18:32:50.327993 17983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e88367b7064efc9dc995574ba0a1d8" candidate_uuid: "2694a3ddafbc484e882015fb46f9bbfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6e2a79583ed46ad9d079b7a72b70b30"
I20260620 18:32:50.328178 18124 raft_consensus.cc:3037] T 10e88367b7064efc9dc995574ba0a1d8 P 4e122b7d3c6f4e4b934949448ccd2fa5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:50.328176 17983 raft_consensus.cc:3037] T 10e88367b7064efc9dc995574ba0a1d8 P f6e2a79583ed46ad9d079b7a72b70b30 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:50.330912 18124 raft_consensus.cc:2454] T 10e88367b7064efc9dc995574ba0a1d8 P 4e122b7d3c6f4e4b934949448ccd2fa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2694a3ddafbc484e882015fb46f9bbfc in term 1.
I20260620 18:32:50.331324 17770 leader_election.cc:302] T 10e88367b7064efc9dc995574ba0a1d8 P 2694a3ddafbc484e882015fb46f9bbfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2694a3ddafbc484e882015fb46f9bbfc, 4e122b7d3c6f4e4b934949448ccd2fa5; no voters:
I20260620 18:32:50.331543 17983 raft_consensus.cc:2454] T 10e88367b7064efc9dc995574ba0a1d8 P f6e2a79583ed46ad9d079b7a72b70b30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2694a3ddafbc484e882015fb46f9bbfc in term 1.
I20260620 18:32:50.331602 18210 raft_consensus.cc:2781] T 10e88367b7064efc9dc995574ba0a1d8 P 2694a3ddafbc484e882015fb46f9bbfc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:50.332146 18210 raft_consensus.cc:686] T 10e88367b7064efc9dc995574ba0a1d8 P 2694a3ddafbc484e882015fb46f9bbfc [term 1 LEADER]: Becoming Leader. State: Replica: 2694a3ddafbc484e882015fb46f9bbfc, State: Running, Role: LEADER
I20260620 18:32:50.332777 18210 consensus_queue.cc:237] T 10e88367b7064efc9dc995574ba0a1d8 P 2694a3ddafbc484e882015fb46f9bbfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e122b7d3c6f4e4b934949448ccd2fa5" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 34005 } } peers { permanent_uuid: "f6e2a79583ed46ad9d079b7a72b70b30" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 35827 } } peers { permanent_uuid: "2694a3ddafbc484e882015fb46f9bbfc" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 45237 } }
I20260620 18:32:50.338943 16917 catalog_manager.cc:5495] T 10e88367b7064efc9dc995574ba0a1d8 P 2694a3ddafbc484e882015fb46f9bbfc reported cstate change: term changed from 0 to 1, leader changed from <none> to 2694a3ddafbc484e882015fb46f9bbfc (127.10.131.1). New cstate: current_term: 1 leader_uuid: "2694a3ddafbc484e882015fb46f9bbfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e122b7d3c6f4e4b934949448ccd2fa5" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 34005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6e2a79583ed46ad9d079b7a72b70b30" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 35827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2694a3ddafbc484e882015fb46f9bbfc" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 45237 } health_report { overall_health: HEALTHY } } }
W20260620 18:32:50.345867 18031 tablet.cc:2369] T 10e88367b7064efc9dc995574ba0a1d8 P f6e2a79583ed46ad9d079b7a72b70b30: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:32:50.379082 18202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.374268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50572 (local address 127.10.131.1:45237)
0620 18:32:50.374457 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:50.374462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:50.374478 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:32:50.374620 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:50.374624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:50.374712 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:50.374833 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:32:50.375129 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.375822 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.377099 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.377476 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.377566 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.377959 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:32:50.378502 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:32:50.378607 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:50.378728 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 18:32:50.378787 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:32:50.387099 18124 raft_consensus.cc:1261] T 10e88367b7064efc9dc995574ba0a1d8 P 4e122b7d3c6f4e4b934949448ccd2fa5 [term 1 FOLLOWER]: Refusing update from remote peer 2694a3ddafbc484e882015fb46f9bbfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:50.387149 17983 raft_consensus.cc:1261] T 10e88367b7064efc9dc995574ba0a1d8 P f6e2a79583ed46ad9d079b7a72b70b30 [term 1 FOLLOWER]: Refusing update from remote peer 2694a3ddafbc484e882015fb46f9bbfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:50.388163 18210 consensus_queue.cc:1035] T 10e88367b7064efc9dc995574ba0a1d8 P 2694a3ddafbc484e882015fb46f9bbfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e122b7d3c6f4e4b934949448ccd2fa5" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 34005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:50.388437 18214 consensus_queue.cc:1035] T 10e88367b7064efc9dc995574ba0a1d8 P 2694a3ddafbc484e882015fb46f9bbfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6e2a79583ed46ad9d079b7a72b70b30" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 35827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:50.402957 18219 mvcc.cc:205] Tried to move back new op lower bound from 7298991597095575552 to 7298991596887646208. Current Snapshot: MvccSnapshot[applied={T|T < 7298991597095575552}]
I20260620 18:32:50.410380 18217 mvcc.cc:205] Tried to move back new op lower bound from 7298991597095575552 to 7298991596887646208. Current Snapshot: MvccSnapshot[applied={T|T < 7298991597095575552}]
I20260620 18:32:50.418082 18218 mvcc.cc:205] Tried to move back new op lower bound from 7298991597095575552 to 7298991596887646208. Current Snapshot: MvccSnapshot[applied={T|T < 7298991597095575552}]
I20260620 18:32:50.429311 16916 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:32:50.429690 16916 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:32:50.438012 16916 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d7e501282a949a1be211a121201e250: Sending DeleteTablet for 3 replicas of tablet 10e88367b7064efc9dc995574ba0a1d8
I20260620 18:32:50.439357 17963 tablet_service.cc:1514] Processing DeleteTablet for tablet 10e88367b7064efc9dc995574ba0a1d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:50 UTC) from {username='oryx', principal='oryx/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:38298
I20260620 18:32:50.439356 18104 tablet_service.cc:1514] Processing DeleteTablet for tablet 10e88367b7064efc9dc995574ba0a1d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:50 UTC) from {username='oryx', principal='oryx/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:52204
I20260620 18:32:50.440346 18228 tablet_replica.cc:330] T 10e88367b7064efc9dc995574ba0a1d8 P 4e122b7d3c6f4e4b934949448ccd2fa5: stopping tablet replica
I20260620 18:32:50.440832 18228 raft_consensus.cc:2229] T 10e88367b7064efc9dc995574ba0a1d8 P 4e122b7d3c6f4e4b934949448ccd2fa5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:32:50.440970 17817 tablet_service.cc:1514] Processing DeleteTablet for tablet 10e88367b7064efc9dc995574ba0a1d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:50 UTC) from {username='oryx', principal='oryx/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:50564
I20260620 18:32:50.441504 18228 raft_consensus.cc:2258] T 10e88367b7064efc9dc995574ba0a1d8 P 4e122b7d3c6f4e4b934949448ccd2fa5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:50.443815 18228 ts_tablet_manager.cc:1905] T 10e88367b7064efc9dc995574ba0a1d8 P 4e122b7d3c6f4e4b934949448ccd2fa5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:50.452606 18228 ts_tablet_manager.cc:1918] T 10e88367b7064efc9dc995574ba0a1d8 P 4e122b7d3c6f4e4b934949448ccd2fa5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:32:50.452811 18229 tablet_replica.cc:330] T 10e88367b7064efc9dc995574ba0a1d8 P 2694a3ddafbc484e882015fb46f9bbfc: stopping tablet replica
I20260620 18:32:50.453416 18229 raft_consensus.cc:2229] T 10e88367b7064efc9dc995574ba0a1d8 P 2694a3ddafbc484e882015fb46f9bbfc [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:32:50.453480 18227 tablet_replica.cc:330] T 10e88367b7064efc9dc995574ba0a1d8 P f6e2a79583ed46ad9d079b7a72b70b30: stopping tablet replica
I20260620 18:32:50.454092 18229 raft_consensus.cc:2258] T 10e88367b7064efc9dc995574ba0a1d8 P 2694a3ddafbc484e882015fb46f9bbfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:50.454113 18227 raft_consensus.cc:2229] T 10e88367b7064efc9dc995574ba0a1d8 P f6e2a79583ed46ad9d079b7a72b70b30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:32:50.454854 18227 raft_consensus.cc:2258] T 10e88367b7064efc9dc995574ba0a1d8 P f6e2a79583ed46ad9d079b7a72b70b30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:50.455415 18228 log.cc:1196] T 10e88367b7064efc9dc995574ba0a1d8 P 4e122b7d3c6f4e4b934949448ccd2fa5: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-2/wal/wals/10e88367b7064efc9dc995574ba0a1d8
I20260620 18:32:50.456259 18228 ts_tablet_manager.cc:1939] T 10e88367b7064efc9dc995574ba0a1d8 P 4e122b7d3c6f4e4b934949448ccd2fa5: Deleting consensus metadata
I20260620 18:32:50.456319 18229 ts_tablet_manager.cc:1905] T 10e88367b7064efc9dc995574ba0a1d8 P 2694a3ddafbc484e882015fb46f9bbfc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:50.457078 18227 ts_tablet_manager.cc:1905] T 10e88367b7064efc9dc995574ba0a1d8 P f6e2a79583ed46ad9d079b7a72b70b30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:50.458694 16901 catalog_manager.cc:4841] TS 4e122b7d3c6f4e4b934949448ccd2fa5 (127.10.131.3:34005): tablet 10e88367b7064efc9dc995574ba0a1d8 (table test-table [id=c28b93e196104fa4bf8287584b8df4fc]) successfully deleted
I20260620 18:32:50.463097 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 16885
I20260620 18:32:50.463505 18229 ts_tablet_manager.cc:1918] T 10e88367b7064efc9dc995574ba0a1d8 P 2694a3ddafbc484e882015fb46f9bbfc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:32:50.463696 18229 log.cc:1196] T 10e88367b7064efc9dc995574ba0a1d8 P 2694a3ddafbc484e882015fb46f9bbfc: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-0/wal/wals/10e88367b7064efc9dc995574ba0a1d8
I20260620 18:32:50.464313 16947 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:50.465076 18227 ts_tablet_manager.cc:1918] T 10e88367b7064efc9dc995574ba0a1d8 P f6e2a79583ed46ad9d079b7a72b70b30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:32:50.465278 18227 log.cc:1196] T 10e88367b7064efc9dc995574ba0a1d8 P f6e2a79583ed46ad9d079b7a72b70b30: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-1/wal/wals/10e88367b7064efc9dc995574ba0a1d8
I20260620 18:32:50.465906 18227 ts_tablet_manager.cc:1939] T 10e88367b7064efc9dc995574ba0a1d8 P f6e2a79583ed46ad9d079b7a72b70b30: Deleting consensus metadata
I20260620 18:32:50.469340 18229 ts_tablet_manager.cc:1939] T 10e88367b7064efc9dc995574ba0a1d8 P 2694a3ddafbc484e882015fb46f9bbfc: Deleting consensus metadata
I20260620 18:32:50.471969 16901 catalog_manager.cc:4841] TS 2694a3ddafbc484e882015fb46f9bbfc (127.10.131.1:45237): tablet 10e88367b7064efc9dc995574ba0a1d8 (table test-table [id=c28b93e196104fa4bf8287584b8df4fc]) successfully deleted
I20260620 18:32:50.480275 18173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.444209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39296 (local address 127.10.131.61:38067)
0620 18:32:50.444375 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:50.444381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:50.461606 (+ 17225us) server_negotiation.cc:407] Connection header received
0620 18:32:50.461803 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:50.461809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:50.461890 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:50.461998 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:50.464578 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.465777 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.467850 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.468564 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.468671 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.472673 (+ 4002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:50.472680 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:50.472684 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:50.472728 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:50.476349 (+ 3621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:50.477313 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:50.477321 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:50.477325 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:50.477469 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:50.477968 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:50.477974 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:50.477977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:50.478366 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:50.478637 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:50.479411 (+ 774us) server_negotiation.cc:299] Negotiation successful
0620 18:32:50.479703 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:32:50.494000 18175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.461759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50902 (local address 127.10.131.60:36193)
0620 18:32:50.461931 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:50.461937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:50.480044 (+ 18107us) server_negotiation.cc:407] Connection header received
0620 18:32:50.480324 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:50.480332 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:50.480428 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:50.480549 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:50.482559 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.483530 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.484904 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.485334 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.485488 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.488163 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:50.488169 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:50.488172 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:50.488207 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:50.490968 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:50.491693 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:50.491700 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:50.491704 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:50.491900 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:50.492298 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:50.492305 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:50.492309 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:50.492615 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:50.492903 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:50.493274 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 18:32:50.493491 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:32:50.666558 16902 catalog_manager.cc:4841] TS f6e2a79583ed46ad9d079b7a72b70b30 (127.10.131.2:35827): tablet 10e88367b7064efc9dc995574ba0a1d8 (table test-table [id=c28b93e196104fa4bf8287584b8df4fc]) successfully deleted
W20260620 18:32:50.666852 16902 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.10.131.2:35827, user_credentials={real_user=slave}} blocked reactor thread for 198686us
I20260620 18:32:50.667508 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 16885
I20260620 18:32:50.721180 18175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.693511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50914 (local address 127.10.131.60:36193)
0620 18:32:50.693704 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:50.693711 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:50.705564 (+ 11853us) server_negotiation.cc:407] Connection header received
0620 18:32:50.705900 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:50.705908 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:50.706004 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:50.706082 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:50.708350 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.709497 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.711442 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.712101 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.712228 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.715329 (+ 3101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:50.715340 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:50.715343 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:50.715374 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:50.717995 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:50.718882 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:50.718889 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:50.718893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:50.719039 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:50.719406 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:50.719414 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:50.719420 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:50.719818 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:50.720039 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:50.720561 (+ 522us) server_negotiation.cc:299] Negotiation successful
0620 18:32:50.720801 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:32:50.722083 18173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:50.693240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39308 (local address 127.10.131.61:38067)
0620 18:32:50.693534 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:50.693542 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:50.706036 (+ 12494us) server_negotiation.cc:407] Connection header received
0620 18:32:50.706437 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:50.706444 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:50.706538 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:50.706657 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:50.708651 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.709503 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.710795 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:50.711191 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:50.711323 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:50.714883 (+ 3560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:50.714890 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:50.714895 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:50.714934 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:50.718198 (+ 3264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:50.719470 (+ 1272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:50.719476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:50.719479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:50.719606 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:50.720019 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:50.720024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:50.720027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:50.720432 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:50.720632 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:50.721520 (+ 888us) server_negotiation.cc:299] Negotiation successful
0620 18:32:50.721751 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260620 18:32:51.414492 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:51.413652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.10.131.2:54405)
0620 18:32:51.414135 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:51.414291 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:44649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":217,"threads_started":1}
W20260620 18:32:51.414994 18029 heartbeater.cc:644] Failed to heartbeat to 127.10.131.62:44649 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.131.62:44649: connect: Connection refused (error 111)
I20260620 18:32:51.420444 18240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:51.419802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.10.131.1:33877)
0620 18:32:51.420186 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:51.420281 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:44649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":205,"threads_started":1}
W20260620 18:32:51.420804 17883 heartbeater.cc:644] Failed to heartbeat to 127.10.131.62:44649 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.131.62:44649: connect: Connection refused (error 111)
I20260620 18:32:51.424319 18241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:51.423496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.10.131.3:55701)
0620 18:32:51.423919 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:51.424064 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:44649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":281,"threads_started":1}
W20260620 18:32:51.424701 18170 heartbeater.cc:644] Failed to heartbeat to 127.10.131.62:44649 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.131.62:44649: connect: Connection refused (error 111)
I20260620 18:32:51.938498 18242 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d7e501282a949a1be211a121201e250)
I20260620 18:32:51.938548 18243 raft_consensus.cc:482] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d7e501282a949a1be211a121201e250)
I20260620 18:32:51.938700 18242 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: false } } peers { permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36193 } attrs { promote: false } }
I20260620 18:32:51.938823 18243 raft_consensus.cc:504] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: false } } peers { permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36193 } attrs { promote: false } }
I20260620 18:32:51.939553 18242 leader_election.cc:288] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2d7e501282a949a1be211a121201e250 (127.10.131.62:44649), 888f00220e5f44c19e0e7f0bff6a1ca9 (127.10.131.61:38067)
I20260620 18:32:51.948346 18243 leader_election.cc:288] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2d7e501282a949a1be211a121201e250 (127.10.131.62:44649), 1d5bf8ef8b624ecfa578fc93e5492446 (127.10.131.60:36193)
I20260620 18:32:51.953541 18248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:51.940510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.0.0.1:46602)
0620 18:32:51.952662 (+ 12152us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:51.953216 (+ 554us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:44649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11973,"threads_started":1}
W20260620 18:32:51.955192 17502 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.131.62:44649: connect: Connection refused (error 111)
I20260620 18:32:51.955454 18245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:51.939724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.0.0.1:46588)
0620 18:32:51.954892 (+ 15168us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:51.955163 (+ 271us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:44649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15030,"threads_started":1}
W20260620 18:32:51.955757 17690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.131.62:44649: connect: Connection refused (error 111)
I20260620 18:32:51.957086 18248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:51.956691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.0.0.1:46630)
0620 18:32:51.956870 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:51.956953 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:44649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 18:32:51.957376 17502 leader_election.cc:334] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d7e501282a949a1be211a121201e250 (127.10.131.62:44649): Network error: Client connection negotiation failed: client connection to 127.10.131.62:44649: connect: Connection refused (error 111)
I20260620 18:32:51.958600 18245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:51.958201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.0.0.1:46644)
0620 18:32:51.958352 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:51.958439 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:44649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 18:32:51.959257 17690 leader_election.cc:334] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d7e501282a949a1be211a121201e250 (127.10.131.62:44649): Network error: Client connection negotiation failed: client connection to 127.10.131.62:44649: connect: Connection refused (error 111)
I20260620 18:32:51.963575 18244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:51.939756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:38067 (local address 127.0.0.1:39314)
0620 18:32:51.955016 (+ 15260us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:51.955564 (+ 548us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:51.955705 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:51.956641 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:51.956646 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:51.956673 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:51.957066 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:51.957078 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:51.959643 (+ 2565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:51.959651 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:51.961132 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:51.961147 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:51.962705 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:51.962711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:51.962917 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:51.963012 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 18:32:51.963085 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 18:32:51.963258 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15111,"mutex_wait_us":91,"thread_start_us":15304,"threads_started":1}
I20260620 18:32:51.963654 18246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:51.940004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39314 (local address 127.10.131.61:38067)
0620 18:32:51.955299 (+ 15295us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:51.955305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:51.956285 (+ 980us) server_negotiation.cc:407] Connection header received
0620 18:32:51.956381 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:51.956394 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:51.956487 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:51.956640 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:51.958277 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:51.959495 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:51.961306 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:51.962535 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:51.962630 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:51.962697 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:51.963159 (+ 462us) server_negotiation.cc:299] Negotiation successful
0620 18:32:51.963309 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":15180,"thread_start_us":15259,"threads_started":1}
I20260620 18:32:51.964254 17525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" is_pre_election: true
I20260620 18:32:51.964569 17525 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d5bf8ef8b624ecfa578fc93e5492446 in term 1.
I20260620 18:32:51.965292 18247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:51.940588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:36193 (local address 127.0.0.1:50926)
0620 18:32:51.954169 (+ 13581us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:51.954186 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:51.954332 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:51.958748 (+ 4416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:51.958752 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:51.958776 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:51.959192 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:51.959202 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:51.961116 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:51.961121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:51.962762 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:51.962775 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:51.964450 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:51.964456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:51.964641 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:51.964682 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 18:32:51.964770 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 18:32:51.964924 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13473,"mutex_wait_us":101,"thread_start_us":12319,"threads_started":1}
I20260620 18:32:51.965181 17691 leader_election.cc:302] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d5bf8ef8b624ecfa578fc93e5492446, 888f00220e5f44c19e0e7f0bff6a1ca9; no voters: 2d7e501282a949a1be211a121201e250
I20260620 18:32:51.965559 18242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 18:32:51.965682 18242 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2d7e501282a949a1be211a121201e250)
I20260620 18:32:51.965773 18249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:51.955309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50926 (local address 127.10.131.60:36193)
0620 18:32:51.958381 (+ 3072us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:51.958387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:51.958405 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:32:51.958470 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:51.958475 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:51.958600 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:51.959442 (+ 842us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:51.959715 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:51.960978 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:51.962920 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:51.964256 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:51.964350 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:51.964466 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:51.965029 (+ 563us) server_negotiation.cc:299] Negotiation successful
0620 18:32:51.965171 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":2947,"thread_start_us":2792,"threads_started":1}
I20260620 18:32:51.965845 18242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:32:51.967873 18242 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: false } } peers { permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36193 } attrs { promote: false } }
I20260620 18:32:51.968506 18242 leader_election.cc:288] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [CANDIDATE]: Term 2 election: Requested vote from peers 2d7e501282a949a1be211a121201e250 (127.10.131.62:44649), 888f00220e5f44c19e0e7f0bff6a1ca9 (127.10.131.61:38067)
I20260620 18:32:51.968803 17714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" is_pre_election: true
I20260620 18:32:51.969177 17714 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 888f00220e5f44c19e0e7f0bff6a1ca9 in current term 2: Already voted for candidate 1d5bf8ef8b624ecfa578fc93e5492446 in this term.
I20260620 18:32:51.969204 18244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:51.968693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.0.0.1:46652)
0620 18:32:51.968903 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:51.969055 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:44649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 18:32:51.969139 17525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9"
I20260620 18:32:51.969340 17525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:32:51.969851 17500 leader_election.cc:302] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 888f00220e5f44c19e0e7f0bff6a1ca9; no voters: 1d5bf8ef8b624ecfa578fc93e5492446, 2d7e501282a949a1be211a121201e250
I20260620 18:32:51.970377 18244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:51.969932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.0.0.1:46660)
0620 18:32:51.970142 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:51.970235 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:44649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260620 18:32:51.970510 17690 leader_election.cc:334] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2d7e501282a949a1be211a121201e250 (127.10.131.62:44649): Network error: Client connection negotiation failed: client connection to 127.10.131.62:44649: connect: Connection refused (error 111)
I20260620 18:32:51.971943 17525 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d5bf8ef8b624ecfa578fc93e5492446 in term 2.
I20260620 18:32:51.972114 18243 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 18:32:51.972364 17691 leader_election.cc:302] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d5bf8ef8b624ecfa578fc93e5492446, 888f00220e5f44c19e0e7f0bff6a1ca9; no voters: 2d7e501282a949a1be211a121201e250
I20260620 18:32:51.972632 18242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 18:32:51.973137 18242 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [term 2 LEADER]: Becoming Leader. State: Replica: 1d5bf8ef8b624ecfa578fc93e5492446, State: Running, Role: LEADER
I20260620 18:32:51.973510 18242 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: false } } peers { permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36193 } attrs { promote: false } }
I20260620 18:32:51.975252 18252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d5bf8ef8b624ecfa578fc93e5492446. Latest consensus state: current_term: 2 leader_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: false } } peers { permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36193 } attrs { promote: false } } }
I20260620 18:32:51.975526 18252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:51.976137 18254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:32:51.979745 18254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:32:51.980492 18254 catalog_manager.cc:1259] Loaded cluster ID: e971777715374237a5b255fc27e8cc4a
I20260620 18:32:51.980561 18254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:32:51.981664 18254 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:32:51.982447 18254 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446: Loaded TSK: 0
I20260620 18:32:51.983328 18254 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:32:52.015271 17704 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:32:52.021998 17704 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:32:52.025776 18244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.025415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.0.0.1:46676)
0620 18:32:52.025574 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:52.025644 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:44649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 18:32:52.025936 17525 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [term 2 FOLLOWER]: Refusing update from remote peer 1d5bf8ef8b624ecfa578fc93e5492446: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 18:32:52.026700 18252 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [LEADER]: Connected to new peer: Peer: permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 18:32:52.026892 18244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.026426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.0.0.1:46692)
0620 18:32:52.026608 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:52.026708 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:44649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 18:32:52.027101 17690 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 -> Peer 2d7e501282a949a1be211a121201e250 (127.10.131.62:44649): Couldn't send request to peer 2d7e501282a949a1be211a121201e250. Status: Network error: Client connection negotiation failed: client connection to 127.10.131.62:44649: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 18:32:52.029426 18243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d5bf8ef8b624ecfa578fc93e5492446. Latest consensus state: current_term: 2 leader_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: false } } peers { permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36193 } attrs { promote: false } } }
I20260620 18:32:52.029719 18243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:32:52.031169 18242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: false } } peers { permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36193 } attrs { promote: false } } }
I20260620 18:32:52.031458 18242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:52.031893 18243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2d7e501282a949a1be211a121201e250" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44649 } } peers { permanent_uuid: "888f00220e5f44c19e0e7f0bff6a1ca9" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 38067 } attrs { promote: false } } peers { permanent_uuid: "1d5bf8ef8b624ecfa578fc93e5492446" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36193 } attrs { promote: false } } }
I20260620 18:32:52.032159 18243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 888f00220e5f44c19e0e7f0bff6a1ca9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:32:52.050205 18245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.039593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:34005 (local address 127.0.0.1:52208)
0620 18:32:52.039927 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:52.039953 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:52.040056 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:52.041382 (+ 1326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:52.041386 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:52.041408 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:52.041793 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:52.041808 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:52.043615 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:52.043624 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:52.046500 (+ 2876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:52.046522 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:52.049347 (+ 2825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:52.049353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:52.049583 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:52.049614 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 18:32:52.049710 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 18:32:52.049863 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219}
I20260620 18:32:52.050863 18262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.039965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52208 (local address 127.10.131.3:34005)
0620 18:32:52.040600 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:52.040606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:52.040624 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:32:52.041102 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:52.041111 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:52.041223 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:52.041960 (+ 737us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:52.042169 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:52.043461 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:52.046712 (+ 3251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:52.049142 (+ 2430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:52.049277 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:52.049397 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:52.050420 (+ 1023us) server_negotiation.cc:299] Negotiation successful
0620 18:32:52.050581 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":252,"threads_started":1}
I20260620 18:32:52.051612 18104 tablet_service.cc:1467] Processing CreateTablet for tablet a6c63986bb6f4701bf03d1e145be152b (DEFAULT_TABLE table=test-table [id=35a7332fc3f64444bfdfa7c8e7c2c753]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:52.052093 18104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6c63986bb6f4701bf03d1e145be152b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:52.056329 18244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.039221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:45237 (local address 127.0.0.1:50580)
0620 18:32:52.039397 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:52.039419 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:52.039553 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:52.050527 (+ 10974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:52.050533 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:52.050557 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:52.050697 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:52.050708 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:52.052043 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:52.052048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:52.053873 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:52.053886 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:52.055616 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:52.055622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:52.055823 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:52.055861 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 18:32:52.055931 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 18:32:52.056065 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":16}
I20260620 18:32:52.057052 18264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.040564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38306 (local address 127.10.131.2:35827)
0620 18:32:52.049328 (+ 8764us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:52.049338 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:52.049364 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:32:52.049452 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:52.049460 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:52.049559 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:52.049680 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:52.050548 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:52.051637 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:52.054791 (+ 3154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:52.055672 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:52.055836 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:52.055952 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:52.056232 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 18:32:52.056373 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8589,"thread_start_us":8475,"threads_started":1}
I20260620 18:32:52.057056 18263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.039979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:35827 (local address 127.0.0.1:38306)
0620 18:32:52.044760 (+ 4781us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:52.044789 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:52.044956 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:52.049771 (+ 4815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:52.049776 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:52.049801 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:52.050152 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:52.050165 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:52.051792 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:52.051797 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:52.054641 (+ 2844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:52.054653 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:52.055909 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:52.055915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:52.056075 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:52.056098 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 18:32:52.056195 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 18:32:52.056369 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4618,"mutex_wait_us":60,"thread_start_us":3912,"threads_started":1}
I20260620 18:32:52.057693 18261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.039312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50580 (local address 127.10.131.1:45237)
0620 18:32:52.050167 (+ 10855us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:52.050171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:52.050185 (+ 14us) server_negotiation.cc:407] Connection header received
0620 18:32:52.050238 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:52.050242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:52.050353 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:52.050492 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:52.050853 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:52.051875 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:52.054036 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:52.055433 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:52.056583 (+ 1150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:52.057228 (+ 645us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:52.057285 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 18:32:52.057431 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10749,"thread_start_us":10646,"threads_started":1}
I20260620 18:32:52.058414 17817 tablet_service.cc:1467] Processing CreateTablet for tablet a6c63986bb6f4701bf03d1e145be152b (DEFAULT_TABLE table=test-table [id=35a7332fc3f64444bfdfa7c8e7c2c753]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:52.058642 17963 tablet_service.cc:1467] Processing CreateTablet for tablet a6c63986bb6f4701bf03d1e145be152b (DEFAULT_TABLE table=test-table [id=35a7332fc3f64444bfdfa7c8e7c2c753]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:52.058892 17817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6c63986bb6f4701bf03d1e145be152b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:52.059072 17963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6c63986bb6f4701bf03d1e145be152b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:52.064325 18266 tablet_bootstrap.cc:492] T a6c63986bb6f4701bf03d1e145be152b P 2694a3ddafbc484e882015fb46f9bbfc: Bootstrap starting.
I20260620 18:32:52.066690 18266 tablet_bootstrap.cc:654] T a6c63986bb6f4701bf03d1e145be152b P 2694a3ddafbc484e882015fb46f9bbfc: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:52.068221 18267 tablet_bootstrap.cc:492] T a6c63986bb6f4701bf03d1e145be152b P f6e2a79583ed46ad9d079b7a72b70b30: Bootstrap starting.
I20260620 18:32:52.069407 18265 tablet_bootstrap.cc:492] T a6c63986bb6f4701bf03d1e145be152b P 4e122b7d3c6f4e4b934949448ccd2fa5: Bootstrap starting.
I20260620 18:32:52.069854 18267 tablet_bootstrap.cc:654] T a6c63986bb6f4701bf03d1e145be152b P f6e2a79583ed46ad9d079b7a72b70b30: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:52.070180 18266 tablet_bootstrap.cc:492] T a6c63986bb6f4701bf03d1e145be152b P 2694a3ddafbc484e882015fb46f9bbfc: No bootstrap required, opened a new log
I20260620 18:32:52.070329 18266 ts_tablet_manager.cc:1397] T a6c63986bb6f4701bf03d1e145be152b P 2694a3ddafbc484e882015fb46f9bbfc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 18:32:52.071113 18266 raft_consensus.cc:348] T a6c63986bb6f4701bf03d1e145be152b P 2694a3ddafbc484e882015fb46f9bbfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2694a3ddafbc484e882015fb46f9bbfc" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 45237 } } peers { permanent_uuid: "4e122b7d3c6f4e4b934949448ccd2fa5" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 34005 } } peers { permanent_uuid: "f6e2a79583ed46ad9d079b7a72b70b30" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 35827 } }
I20260620 18:32:52.071310 18266 raft_consensus.cc:374] T a6c63986bb6f4701bf03d1e145be152b P 2694a3ddafbc484e882015fb46f9bbfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:52.071398 18266 raft_consensus.cc:729] T a6c63986bb6f4701bf03d1e145be152b P 2694a3ddafbc484e882015fb46f9bbfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2694a3ddafbc484e882015fb46f9bbfc, State: Initialized, Role: FOLLOWER
I20260620 18:32:52.071632 18265 tablet_bootstrap.cc:654] T a6c63986bb6f4701bf03d1e145be152b P 4e122b7d3c6f4e4b934949448ccd2fa5: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:52.071660 18266 consensus_queue.cc:260] T a6c63986bb6f4701bf03d1e145be152b P 2694a3ddafbc484e882015fb46f9bbfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2694a3ddafbc484e882015fb46f9bbfc" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 45237 } } peers { permanent_uuid: "4e122b7d3c6f4e4b934949448ccd2fa5" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 34005 } } peers { permanent_uuid: "f6e2a79583ed46ad9d079b7a72b70b30" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 35827 } }
I20260620 18:32:52.075758 18266 ts_tablet_manager.cc:1428] T a6c63986bb6f4701bf03d1e145be152b P 2694a3ddafbc484e882015fb46f9bbfc: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260620 18:32:52.079062 17882 heartbeater.cc:503] Master 127.10.131.60:36193 was elected leader, sending a full tablet report...
I20260620 18:32:52.093379 18272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.081240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.10.131.1:49375)
0620 18:32:52.092779 (+ 11539us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:52.093006 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:44649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11362,"thread_start_us":11466,"threads_started":1}
I20260620 18:32:52.094553 18267 tablet_bootstrap.cc:492] T a6c63986bb6f4701bf03d1e145be152b P f6e2a79583ed46ad9d079b7a72b70b30: No bootstrap required, opened a new log
I20260620 18:32:52.094753 18267 ts_tablet_manager.cc:1397] T a6c63986bb6f4701bf03d1e145be152b P f6e2a79583ed46ad9d079b7a72b70b30: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.012s
I20260620 18:32:52.095825 18267 raft_consensus.cc:348] T a6c63986bb6f4701bf03d1e145be152b P f6e2a79583ed46ad9d079b7a72b70b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2694a3ddafbc484e882015fb46f9bbfc" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 45237 } } peers { permanent_uuid: "4e122b7d3c6f4e4b934949448ccd2fa5" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 34005 } } peers { permanent_uuid: "f6e2a79583ed46ad9d079b7a72b70b30" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 35827 } }
I20260620 18:32:52.096159 18267 raft_consensus.cc:374] T a6c63986bb6f4701bf03d1e145be152b P f6e2a79583ed46ad9d079b7a72b70b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:52.096283 18267 raft_consensus.cc:729] T a6c63986bb6f4701bf03d1e145be152b P f6e2a79583ed46ad9d079b7a72b70b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6e2a79583ed46ad9d079b7a72b70b30, State: Initialized, Role: FOLLOWER
I20260620 18:32:52.096609 18267 consensus_queue.cc:260] T a6c63986bb6f4701bf03d1e145be152b P f6e2a79583ed46ad9d079b7a72b70b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2694a3ddafbc484e882015fb46f9bbfc" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 45237 } } peers { permanent_uuid: "4e122b7d3c6f4e4b934949448ccd2fa5" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 34005 } } peers { permanent_uuid: "f6e2a79583ed46ad9d079b7a72b70b30" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 35827 } }
I20260620 18:32:52.098845 18265 tablet_bootstrap.cc:492] T a6c63986bb6f4701bf03d1e145be152b P 4e122b7d3c6f4e4b934949448ccd2fa5: No bootstrap required, opened a new log
I20260620 18:32:52.098999 18265 ts_tablet_manager.cc:1397] T a6c63986bb6f4701bf03d1e145be152b P 4e122b7d3c6f4e4b934949448ccd2fa5: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.020s
I20260620 18:32:52.099897 18272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.097624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.10.131.1:56971)
0620 18:32:52.098052 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:52.099725 (+ 1673us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:44649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330}
I20260620 18:32:52.099817 18265 raft_consensus.cc:348] T a6c63986bb6f4701bf03d1e145be152b P 4e122b7d3c6f4e4b934949448ccd2fa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2694a3ddafbc484e882015fb46f9bbfc" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 45237 } } peers { permanent_uuid: "4e122b7d3c6f4e4b934949448ccd2fa5" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 34005 } } peers { permanent_uuid: "f6e2a79583ed46ad9d079b7a72b70b30" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 35827 } }
I20260620 18:32:52.100080 18265 raft_consensus.cc:374] T a6c63986bb6f4701bf03d1e145be152b P 4e122b7d3c6f4e4b934949448ccd2fa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:52.100167 18265 raft_consensus.cc:729] T a6c63986bb6f4701bf03d1e145be152b P 4e122b7d3c6f4e4b934949448ccd2fa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e122b7d3c6f4e4b934949448ccd2fa5, State: Initialized, Role: FOLLOWER
I20260620 18:32:52.100412 18267 ts_tablet_manager.cc:1428] T a6c63986bb6f4701bf03d1e145be152b P f6e2a79583ed46ad9d079b7a72b70b30: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260620 18:32:52.100395 18265 consensus_queue.cc:260] T a6c63986bb6f4701bf03d1e145be152b P 4e122b7d3c6f4e4b934949448ccd2fa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2694a3ddafbc484e882015fb46f9bbfc" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 45237 } } peers { permanent_uuid: "4e122b7d3c6f4e4b934949448ccd2fa5" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 34005 } } peers { permanent_uuid: "f6e2a79583ed46ad9d079b7a72b70b30" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 35827 } }
I20260620 18:32:52.100708 18028 heartbeater.cc:503] Master 127.10.131.60:36193 was elected leader, sending a full tablet report...
I20260620 18:32:52.103294 18265 ts_tablet_manager.cc:1428] T a6c63986bb6f4701bf03d1e145be152b P 4e122b7d3c6f4e4b934949448ccd2fa5: Time spent starting tablet: real 0.004s user 0.000s sys 0.005s
I20260620 18:32:52.103421 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.097995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.10.131.2:60099)
0620 18:32:52.102971 (+ 4976us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:52.103108 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:44649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4810,"thread_start_us":4644,"threads_started":1}
I20260620 18:32:52.104692 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.103484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.10.131.3:34599)
0620 18:32:52.104311 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:52.104462 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:44649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":347,"threads_started":1}
I20260620 18:32:52.104699 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.104265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.10.131.2:60545)
0620 18:32:52.104478 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:52.104557 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:44649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260620 18:32:52.105219 18169 heartbeater.cc:503] Master 127.10.131.60:36193 was elected leader, sending a full tablet report...
I20260620 18:32:52.105808 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.105457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.10.131.3:51645)
0620 18:32:52.105614 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:52.105698 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:44649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 18:32:52.194629 18275 raft_consensus.cc:482] T a6c63986bb6f4701bf03d1e145be152b P 4e122b7d3c6f4e4b934949448ccd2fa5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:32:52.194873 18275 raft_consensus.cc:504] T a6c63986bb6f4701bf03d1e145be152b P 4e122b7d3c6f4e4b934949448ccd2fa5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2694a3ddafbc484e882015fb46f9bbfc" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 45237 } } peers { permanent_uuid: "4e122b7d3c6f4e4b934949448ccd2fa5" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 34005 } } peers { permanent_uuid: "f6e2a79583ed46ad9d079b7a72b70b30" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 35827 } }
I20260620 18:32:52.196650 18275 leader_election.cc:288] T a6c63986bb6f4701bf03d1e145be152b P 4e122b7d3c6f4e4b934949448ccd2fa5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2694a3ddafbc484e882015fb46f9bbfc (127.10.131.1:45237), f6e2a79583ed46ad9d079b7a72b70b30 (127.10.131.2:35827)
I20260620 18:32:52.203148 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.196823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:45237 (local address 127.10.131.3:38659)
0620 18:32:52.197132 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:52.197156 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:52.197351 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:52.197878 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:52.197886 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:52.197916 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:52.198215 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:52.198240 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:52.199490 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:52.199496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:52.201116 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:52.201128 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:52.202428 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:52.202435 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:52.202616 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:52.202652 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 18:32:52.202755 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 18:32:52.202909 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181}
I20260620 18:32:52.203245 18261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.196969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:38659 (local address 127.10.131.1:45237)
0620 18:32:52.197167 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:52.197173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:52.197248 (+ 75us) server_negotiation.cc:407] Connection header received
0620 18:32:52.197634 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:52.197639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:52.197725 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:52.197803 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:52.198411 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:52.199362 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:52.201262 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:52.202254 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:52.202361 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:52.202411 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:52.202815 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 18:32:52.202972 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:32:52.203605 18264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.197217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:45779 (local address 127.10.131.2:35827)
0620 18:32:52.197485 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:52.197493 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:52.197917 (+ 424us) server_negotiation.cc:407] Connection header received
0620 18:32:52.198070 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:52.198073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:52.198145 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:52.198231 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:52.198860 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:52.199841 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:52.201826 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:52.202689 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:52.202822 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:52.202887 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:52.203239 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 18:32:52.203369 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 18:32:52.203820 18277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.197065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:35827 (local address 127.10.131.3:45779)
0620 18:32:52.197795 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:52.197815 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:52.197946 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:52.198267 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:52.198274 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:52.198305 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:52.198672 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:52.198685 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:52.200095 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:52.200099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:52.201677 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:52.201692 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:52.202864 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:52.202876 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:52.203066 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:52.203099 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:32:52.203198 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 18:32:52.203501 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":435,"threads_started":1}
I20260620 18:32:52.204144 17837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6c63986bb6f4701bf03d1e145be152b" candidate_uuid: "4e122b7d3c6f4e4b934949448ccd2fa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2694a3ddafbc484e882015fb46f9bbfc" is_pre_election: true
I20260620 18:32:52.204517 17983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6c63986bb6f4701bf03d1e145be152b" candidate_uuid: "4e122b7d3c6f4e4b934949448ccd2fa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6e2a79583ed46ad9d079b7a72b70b30" is_pre_election: true
I20260620 18:32:52.204567 17837 raft_consensus.cc:2454] T a6c63986bb6f4701bf03d1e145be152b P 2694a3ddafbc484e882015fb46f9bbfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e122b7d3c6f4e4b934949448ccd2fa5 in term 0.
I20260620 18:32:52.204732 17983 raft_consensus.cc:2454] T a6c63986bb6f4701bf03d1e145be152b P f6e2a79583ed46ad9d079b7a72b70b30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e122b7d3c6f4e4b934949448ccd2fa5 in term 0.
I20260620 18:32:52.205137 18057 leader_election.cc:302] T a6c63986bb6f4701bf03d1e145be152b P 4e122b7d3c6f4e4b934949448ccd2fa5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2694a3ddafbc484e882015fb46f9bbfc, 4e122b7d3c6f4e4b934949448ccd2fa5; no voters:
I20260620 18:32:52.205492 18275 raft_consensus.cc:2781] T a6c63986bb6f4701bf03d1e145be152b P 4e122b7d3c6f4e4b934949448ccd2fa5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:32:52.205627 18275 raft_consensus.cc:482] T a6c63986bb6f4701bf03d1e145be152b P 4e122b7d3c6f4e4b934949448ccd2fa5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:32:52.205742 18275 raft_consensus.cc:3037] T a6c63986bb6f4701bf03d1e145be152b P 4e122b7d3c6f4e4b934949448ccd2fa5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:52.207768 18275 raft_consensus.cc:504] T a6c63986bb6f4701bf03d1e145be152b P 4e122b7d3c6f4e4b934949448ccd2fa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2694a3ddafbc484e882015fb46f9bbfc" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 45237 } } peers { permanent_uuid: "4e122b7d3c6f4e4b934949448ccd2fa5" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 34005 } } peers { permanent_uuid: "f6e2a79583ed46ad9d079b7a72b70b30" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 35827 } }
I20260620 18:32:52.208305 18275 leader_election.cc:288] T a6c63986bb6f4701bf03d1e145be152b P 4e122b7d3c6f4e4b934949448ccd2fa5 [CANDIDATE]: Term 1 election: Requested vote from peers 2694a3ddafbc484e882015fb46f9bbfc (127.10.131.1:45237), f6e2a79583ed46ad9d079b7a72b70b30 (127.10.131.2:35827)
I20260620 18:32:52.208796 17983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6c63986bb6f4701bf03d1e145be152b" candidate_uuid: "4e122b7d3c6f4e4b934949448ccd2fa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6e2a79583ed46ad9d079b7a72b70b30"
I20260620 18:32:52.209060 17983 raft_consensus.cc:3037] T a6c63986bb6f4701bf03d1e145be152b P f6e2a79583ed46ad9d079b7a72b70b30 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:52.209070 17837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6c63986bb6f4701bf03d1e145be152b" candidate_uuid: "4e122b7d3c6f4e4b934949448ccd2fa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2694a3ddafbc484e882015fb46f9bbfc"
I20260620 18:32:52.209276 17837 raft_consensus.cc:3037] T a6c63986bb6f4701bf03d1e145be152b P 2694a3ddafbc484e882015fb46f9bbfc [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:52.210991 17983 raft_consensus.cc:2454] T a6c63986bb6f4701bf03d1e145be152b P f6e2a79583ed46ad9d079b7a72b70b30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e122b7d3c6f4e4b934949448ccd2fa5 in term 1.
I20260620 18:32:52.211278 17837 raft_consensus.cc:2454] T a6c63986bb6f4701bf03d1e145be152b P 2694a3ddafbc484e882015fb46f9bbfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e122b7d3c6f4e4b934949448ccd2fa5 in term 1.
I20260620 18:32:52.211318 18058 leader_election.cc:302] T a6c63986bb6f4701bf03d1e145be152b P 4e122b7d3c6f4e4b934949448ccd2fa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e122b7d3c6f4e4b934949448ccd2fa5, f6e2a79583ed46ad9d079b7a72b70b30; no voters:
I20260620 18:32:52.211645 18275 raft_consensus.cc:2781] T a6c63986bb6f4701bf03d1e145be152b P 4e122b7d3c6f4e4b934949448ccd2fa5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:52.212066 18275 raft_consensus.cc:686] T a6c63986bb6f4701bf03d1e145be152b P 4e122b7d3c6f4e4b934949448ccd2fa5 [term 1 LEADER]: Becoming Leader. State: Replica: 4e122b7d3c6f4e4b934949448ccd2fa5, State: Running, Role: LEADER
I20260620 18:32:52.212461 18275 consensus_queue.cc:237] T a6c63986bb6f4701bf03d1e145be152b P 4e122b7d3c6f4e4b934949448ccd2fa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2694a3ddafbc484e882015fb46f9bbfc" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 45237 } } peers { permanent_uuid: "4e122b7d3c6f4e4b934949448ccd2fa5" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 34005 } } peers { permanent_uuid: "f6e2a79583ed46ad9d079b7a72b70b30" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 35827 } }
I20260620 18:32:52.213034 18277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.212592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.10.131.3:46983)
0620 18:32:52.212731 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:52.212897 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:44649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 18:32:52.215709 17704 catalog_manager.cc:5495] T a6c63986bb6f4701bf03d1e145be152b P 4e122b7d3c6f4e4b934949448ccd2fa5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e122b7d3c6f4e4b934949448ccd2fa5 (127.10.131.3). New cstate: current_term: 1 leader_uuid: "4e122b7d3c6f4e4b934949448ccd2fa5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2694a3ddafbc484e882015fb46f9bbfc" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e122b7d3c6f4e4b934949448ccd2fa5" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 34005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6e2a79583ed46ad9d079b7a72b70b30" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 35827 } health_report { overall_health: UNKNOWN } } }
I20260620 18:32:52.246587 18262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.241984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52222 (local address 127.10.131.3:34005)
0620 18:32:52.242162 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:52.242167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:52.242185 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:32:52.242385 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:52.242390 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:52.242473 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:52.242592 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:32:52.242931 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:52.243598 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:52.244778 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:52.245234 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:52.245325 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:52.245668 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:32:52.246165 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:32:52.246272 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:52.246343 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 18:32:52.246399 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 18:32:52.250391 17983 raft_consensus.cc:1261] T a6c63986bb6f4701bf03d1e145be152b P f6e2a79583ed46ad9d079b7a72b70b30 [term 1 FOLLOWER]: Refusing update from remote peer 4e122b7d3c6f4e4b934949448ccd2fa5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:52.250730 17837 raft_consensus.cc:1261] T a6c63986bb6f4701bf03d1e145be152b P 2694a3ddafbc484e882015fb46f9bbfc [term 1 FOLLOWER]: Refusing update from remote peer 4e122b7d3c6f4e4b934949448ccd2fa5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:52.251132 18278 consensus_queue.cc:1035] T a6c63986bb6f4701bf03d1e145be152b P 4e122b7d3c6f4e4b934949448ccd2fa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6e2a79583ed46ad9d079b7a72b70b30" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 35827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:52.251488 18275 consensus_queue.cc:1035] T a6c63986bb6f4701bf03d1e145be152b P 4e122b7d3c6f4e4b934949448ccd2fa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2694a3ddafbc484e882015fb46f9bbfc" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:52.254406 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.253967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.10.131.2:36207)
0620 18:32:52.254114 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:52.254218 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:44649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 18:32:52.254446 18272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.253589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.10.131.1:56401)
0620 18:32:52.254193 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:52.254287 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:44649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502}
I20260620 18:32:52.262185 18272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.261688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.10.131.1:54129)
0620 18:32:52.261907 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:52.262004 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:44649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 18:32:52.262626 18277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.262071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.10.131.3:34081)
0620 18:32:52.262321 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:52.262422 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:44649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260620 18:32:52.264416 18277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.263536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.10.131.3:57043)
0620 18:32:52.264167 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:52.264248 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:44649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":533}
I20260620 18:32:52.265275 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.264785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.10.131.2:41265)
0620 18:32:52.265035 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:52.265126 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:44649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260620 18:32:52.274366 17704 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:32:52.274653 17704 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:32:52.279968 17704 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d5bf8ef8b624ecfa578fc93e5492446: Sending DeleteTablet for 3 replicas of tablet a6c63986bb6f4701bf03d1e145be152b
I20260620 18:32:52.280836 17817 tablet_service.cc:1514] Processing DeleteTablet for tablet a6c63986bb6f4701bf03d1e145be152b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:52 UTC) from {username='oryx', principal='oryx/127.10.131.60@KRBTEST.COM'} at 127.0.0.1:50580
I20260620 18:32:52.281505 17963 tablet_service.cc:1514] Processing DeleteTablet for tablet a6c63986bb6f4701bf03d1e145be152b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:52 UTC) from {username='oryx', principal='oryx/127.10.131.60@KRBTEST.COM'} at 127.0.0.1:38306
I20260620 18:32:52.281488 18104 tablet_service.cc:1514] Processing DeleteTablet for tablet a6c63986bb6f4701bf03d1e145be152b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:52 UTC) from {username='oryx', principal='oryx/127.10.131.60@KRBTEST.COM'} at 127.0.0.1:52208
I20260620 18:32:52.287933 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 17676
I20260620 18:32:52.293391 18292 tablet_replica.cc:330] T a6c63986bb6f4701bf03d1e145be152b P f6e2a79583ed46ad9d079b7a72b70b30: stopping tablet replica
I20260620 18:32:52.293628 18292 raft_consensus.cc:2229] T a6c63986bb6f4701bf03d1e145be152b P f6e2a79583ed46ad9d079b7a72b70b30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:32:52.293946 18292 raft_consensus.cc:2258] T a6c63986bb6f4701bf03d1e145be152b P f6e2a79583ed46ad9d079b7a72b70b30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:52.295032 18292 ts_tablet_manager.cc:1905] T a6c63986bb6f4701bf03d1e145be152b P f6e2a79583ed46ad9d079b7a72b70b30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:52.301719 18292 ts_tablet_manager.cc:1918] T a6c63986bb6f4701bf03d1e145be152b P f6e2a79583ed46ad9d079b7a72b70b30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:32:52.301872 18292 log.cc:1196] T a6c63986bb6f4701bf03d1e145be152b P f6e2a79583ed46ad9d079b7a72b70b30: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-1/wal/wals/a6c63986bb6f4701bf03d1e145be152b
I20260620 18:32:52.302359 18292 ts_tablet_manager.cc:1939] T a6c63986bb6f4701bf03d1e145be152b P f6e2a79583ed46ad9d079b7a72b70b30: Deleting consensus metadata
I20260620 18:32:52.303505 17689 catalog_manager.cc:4841] TS f6e2a79583ed46ad9d079b7a72b70b30 (127.10.131.2:35827): tablet a6c63986bb6f4701bf03d1e145be152b (table test-table [id=35a7332fc3f64444bfdfa7c8e7c2c753]) successfully deleted
I20260620 18:32:52.305349 18293 tablet_replica.cc:330] T a6c63986bb6f4701bf03d1e145be152b P 4e122b7d3c6f4e4b934949448ccd2fa5: stopping tablet replica
I20260620 18:32:52.305548 18293 raft_consensus.cc:2229] T a6c63986bb6f4701bf03d1e145be152b P 4e122b7d3c6f4e4b934949448ccd2fa5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:32:52.306174 18293 raft_consensus.cc:2258] T a6c63986bb6f4701bf03d1e145be152b P 4e122b7d3c6f4e4b934949448ccd2fa5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:52.307030 18291 tablet_replica.cc:330] T a6c63986bb6f4701bf03d1e145be152b P 2694a3ddafbc484e882015fb46f9bbfc: stopping tablet replica
I20260620 18:32:52.307232 18291 raft_consensus.cc:2229] T a6c63986bb6f4701bf03d1e145be152b P 2694a3ddafbc484e882015fb46f9bbfc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:32:52.307531 18291 raft_consensus.cc:2258] T a6c63986bb6f4701bf03d1e145be152b P 2694a3ddafbc484e882015fb46f9bbfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:52.308300 18293 ts_tablet_manager.cc:1905] T a6c63986bb6f4701bf03d1e145be152b P 4e122b7d3c6f4e4b934949448ccd2fa5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:52.308724 18291 ts_tablet_manager.cc:1905] T a6c63986bb6f4701bf03d1e145be152b P 2694a3ddafbc484e882015fb46f9bbfc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:52.315052 18293 ts_tablet_manager.cc:1918] T a6c63986bb6f4701bf03d1e145be152b P 4e122b7d3c6f4e4b934949448ccd2fa5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:32:52.315197 18293 log.cc:1196] T a6c63986bb6f4701bf03d1e145be152b P 4e122b7d3c6f4e4b934949448ccd2fa5: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-2/wal/wals/a6c63986bb6f4701bf03d1e145be152b
I20260620 18:32:52.315544 18291 ts_tablet_manager.cc:1918] T a6c63986bb6f4701bf03d1e145be152b P 2694a3ddafbc484e882015fb46f9bbfc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:32:52.315687 18293 ts_tablet_manager.cc:1939] T a6c63986bb6f4701bf03d1e145be152b P 4e122b7d3c6f4e4b934949448ccd2fa5: Deleting consensus metadata
I20260620 18:32:52.315729 18291 log.cc:1196] T a6c63986bb6f4701bf03d1e145be152b P 2694a3ddafbc484e882015fb46f9bbfc: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781980314095189-10764-0/minicluster-data/ts-0/wal/wals/a6c63986bb6f4701bf03d1e145be152b
I20260620 18:32:52.316355 18291 ts_tablet_manager.cc:1939] T a6c63986bb6f4701bf03d1e145be152b P 2694a3ddafbc484e882015fb46f9bbfc: Deleting consensus metadata
I20260620 18:32:52.316998 17688 catalog_manager.cc:4841] TS 4e122b7d3c6f4e4b934949448ccd2fa5 (127.10.131.3:34005): tablet a6c63986bb6f4701bf03d1e145be152b (table test-table [id=35a7332fc3f64444bfdfa7c8e7c2c753]) successfully deleted
I20260620 18:32:52.317579 17688 catalog_manager.cc:4841] TS 2694a3ddafbc484e882015fb46f9bbfc (127.10.131.1:45237): tablet a6c63986bb6f4701bf03d1e145be152b (table test-table [id=35a7332fc3f64444bfdfa7c8e7c2c753]) successfully deleted
I20260620 18:32:52.321974 18261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.289005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50594 (local address 127.10.131.1:45237)
0620 18:32:52.289179 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:52.289187 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:52.303161 (+ 13974us) server_negotiation.cc:407] Connection header received
0620 18:32:52.303698 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:52.303708 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:52.303814 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:52.303898 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:52.306412 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:52.307536 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:52.309539 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:52.310176 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:52.310275 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:52.313729 (+ 3454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:52.313745 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:52.313768 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:52.313803 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:52.317730 (+ 3927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:52.318520 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:52.318527 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:52.318529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:52.318640 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:52.319040 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:52.319043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:52.319046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:52.319343 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:52.319553 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:52.321342 (+ 1789us) server_negotiation.cc:299] Negotiation successful
0620 18:32:52.321556 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:32:52.322568 17877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:52.415439 18263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.414937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.0.0.1:46704)
0620 18:32:52.415140 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:52.415257 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:44649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 18:32:52.416554 18263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.416133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.0.0.1:46720)
0620 18:32:52.416322 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:52.416409 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:44649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260620 18:32:52.454761 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 17676
I20260620 18:32:52.477747 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 17894
I20260620 18:32:52.492352 18264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.478546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38322 (local address 127.10.131.2:35827)
0620 18:32:52.478773 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:52.478780 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:52.478802 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:32:52.478954 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:52.478960 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:52.479054 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:52.479162 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:52.480361 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:52.481462 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:52.482838 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:52.483446 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:52.483562 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:52.486010 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:52.486027 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:52.486043 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:52.486081 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:52.489569 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:52.490243 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:52.490252 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:52.490256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:52.490445 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:52.490853 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:52.490858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:52.490862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:52.491292 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:52.491577 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:52.491765 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 18:32:52.491993 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260620 18:32:52.493059 18023 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:52.637028 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 17894
I20260620 18:32:52.654606 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 18041
I20260620 18:32:52.670676 18262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.655523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52232 (local address 127.10.131.3:34005)
0620 18:32:52.655724 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:52.655730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:52.655752 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:32:52.655825 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:52.655831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:52.655935 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:52.656045 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:52.657518 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:52.658802 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:52.660188 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:52.660963 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:52.661104 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:52.663554 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:52.663577 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:52.663599 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:52.663640 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:52.667800 (+ 4160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:52.668448 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:52.668457 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:52.668461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:52.668636 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:52.669044 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:52.669050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:52.669054 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:52.669547 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:52.669874 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:52.670043 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 18:32:52.670269 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:32:52.671512 18164 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:52.800774 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 18041
I20260620 18:32:52.822064 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 17378
I20260620 18:32:52.823125 17546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:52.936507 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 17378
I20260620 18:32:52.942723 18299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.942019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.0.0.1:46722)
0620 18:32:52.942467 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:52.942573 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:44649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":252,"threads_started":1}
I20260620 18:32:52.953958 18299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:52.953602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:44649 (local address 127.0.0.1:46738)
0620 18:32:52.953754 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:52.953835 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:44649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 18:32:52.957346 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 17480
I20260620 18:32:52.958287 17734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:53.075606 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 17480
2026-06-20T18:32:53Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9627 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:32:53 dist-test-slave-cbn3 krb5kdc[18304](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:32:53 dist-test-slave-cbn3 krb5kdc[18304](info): set up 2 sockets
Jun 20 18:32:53 dist-test-slave-cbn3 krb5kdc[18304](info): commencing operation
krb5kdc: starting...
W20260620 18:32:54.165832 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:32:54 dist-test-slave-cbn3 krb5kdc[18304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:32:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:32:54Z Disabled control of system clock
WARNING: no policy specified for oryx/127.10.131.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.131.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.62.keytab.
Entry for principal oryx/127.10.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.62.keytab.
I20260620 18:32:54.373903 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:40381
--webserver_interface=127.10.131.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.62.keytab
--principal=oryx/127.10.131.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:46739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:40381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:54.690379 18317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:54.690692 18317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:54.690845 18317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:54.697809 18317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:32:54.697945 18317 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 18:32:54.698035 18317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:54.698150 18317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:54.698237 18317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:32:54.698343 18317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:32:54.708096 18317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:46739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:40381
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.62.keytab
--principal=oryx/127.10.131.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:40381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:54.709360 18317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:54.710561 18317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:54.721714 18325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:54.721807 18322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:54.722172 18323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:54.722635 18317 server_base.cc:1034] running on GCE node
I20260620 18:32:54.723332 18317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:54.724613 18317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:54.725848 18317 hybrid_clock.cc:648] HybridClock initialized: now 1781980374725772 us; error 53 us; skew 500 ppm
I20260620 18:32:54.726159 18317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:54 dist-test-slave-cbn3 krb5kdc[18304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980374, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:54.729261 18317 init.cc:377] Logged in from keytab as oryx/127.10.131.62@KRBTEST.COM (short username oryx)
I20260620 18:32:54.730562 18317 webserver.cc:460] Webserver started at http://127.10.131.62:38473/ using document root <none> and password file <none>
I20260620 18:32:54.731240 18317 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:54.731367 18317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:54.731678 18317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:54.734546 18317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "012beec4185d4c8484308e45c3301e34"
format_stamp: "Formatted at 2026-06-20 18:32:54 on dist-test-slave-cbn3"
server_key: "88d83062a5dc0b8074f29ead18f68a5b"
server_key_iv: "aee361c0833c9af3026bcd62d96a38fc"
server_key_version: "encryptionkey@0"
I20260620 18:32:54.735350 18317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "012beec4185d4c8484308e45c3301e34"
format_stamp: "Formatted at 2026-06-20 18:32:54 on dist-test-slave-cbn3"
server_key: "88d83062a5dc0b8074f29ead18f68a5b"
server_key_iv: "aee361c0833c9af3026bcd62d96a38fc"
server_key_version: "encryptionkey@0"
I20260620 18:32:54.740206 18317 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 18:32:54.743464 18332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:54.744699 18317 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 18:32:54.744956 18317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "012beec4185d4c8484308e45c3301e34"
format_stamp: "Formatted at 2026-06-20 18:32:54 on dist-test-slave-cbn3"
server_key: "88d83062a5dc0b8074f29ead18f68a5b"
server_key_iv: "aee361c0833c9af3026bcd62d96a38fc"
server_key_version: "encryptionkey@0"
I20260620 18:32:54.745163 18317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:54.761694 18317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:54.766083 18317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:54.766359 18317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:54.787326 18317 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:40381
I20260620 18:32:54.787344 18393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:40381 every 8 connection(s)
I20260620 18:32:54.788609 18317 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:32:54.792140 18394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:54.792827 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 18317
I20260620 18:32:54.793095 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:32:54.793725 10764 external_mini_cluster.cc:1428] Setting key a2f21a488ff621aa5ed8b48732dca071
I20260620 18:32:54.805425 18394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 012beec4185d4c8484308e45c3301e34: Bootstrap starting.
Jun 20 18:32:54 dist-test-slave-cbn3 krb5kdc[18304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM
I20260620 18:32:54.809008 18394 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 012beec4185d4c8484308e45c3301e34: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:54.810086 18394 log.cc:824] T 00000000000000000000000000000000 P 012beec4185d4c8484308e45c3301e34: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:54.812829 18394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 012beec4185d4c8484308e45c3301e34: No bootstrap required, opened a new log
I20260620 18:32:54.815790 18397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:54.795554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50920 (local address 127.10.131.62:40381)
0620 18:32:54.796348 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:54.796358 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:54.796397 (+ 39us) server_negotiation.cc:407] Connection header received
0620 18:32:54.798013 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:54.798033 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:54.798404 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:54.798710 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:54.800365 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:54.801648 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:54.803405 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:54.804008 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:54.804128 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:54.808634 (+ 4506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:54.808642 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:54.808656 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:54.808700 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:54.811800 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:54.812500 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:54.812506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:54.812512 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:54.812678 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:54.813110 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:54.813116 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:54.813120 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:54.813630 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:54.813934 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:54.814388 (+ 454us) server_negotiation.cc:299] Negotiation successful
0620 18:32:54.814807 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":381,"threads_started":1}
I20260620 18:32:54.821146 18394 raft_consensus.cc:348] T 00000000000000000000000000000000 P 012beec4185d4c8484308e45c3301e34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012beec4185d4c8484308e45c3301e34" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 40381 } }
I20260620 18:32:54.821494 18394 raft_consensus.cc:374] T 00000000000000000000000000000000 P 012beec4185d4c8484308e45c3301e34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:54.821594 18394 raft_consensus.cc:729] T 00000000000000000000000000000000 P 012beec4185d4c8484308e45c3301e34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 012beec4185d4c8484308e45c3301e34, State: Initialized, Role: FOLLOWER
I20260620 18:32:54.822088 18394 consensus_queue.cc:260] T 00000000000000000000000000000000 P 012beec4185d4c8484308e45c3301e34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012beec4185d4c8484308e45c3301e34" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 40381 } }
I20260620 18:32:54.822366 18394 raft_consensus.cc:388] T 00000000000000000000000000000000 P 012beec4185d4c8484308e45c3301e34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:32:54.822512 18394 raft_consensus.cc:482] T 00000000000000000000000000000000 P 012beec4185d4c8484308e45c3301e34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:32:54.822731 18394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 012beec4185d4c8484308e45c3301e34 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:54.825001 18394 raft_consensus.cc:504] T 00000000000000000000000000000000 P 012beec4185d4c8484308e45c3301e34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012beec4185d4c8484308e45c3301e34" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 40381 } }
I20260620 18:32:54.825465 18394 leader_election.cc:302] T 00000000000000000000000000000000 P 012beec4185d4c8484308e45c3301e34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 012beec4185d4c8484308e45c3301e34; no voters:
I20260620 18:32:54.826114 18394 leader_election.cc:288] T 00000000000000000000000000000000 P 012beec4185d4c8484308e45c3301e34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:32:54.826265 18399 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 012beec4185d4c8484308e45c3301e34 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:54.827039 18399 raft_consensus.cc:686] T 00000000000000000000000000000000 P 012beec4185d4c8484308e45c3301e34 [term 1 LEADER]: Becoming Leader. State: Replica: 012beec4185d4c8484308e45c3301e34, State: Running, Role: LEADER
I20260620 18:32:54.827566 18399 consensus_queue.cc:237] T 00000000000000000000000000000000 P 012beec4185d4c8484308e45c3301e34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012beec4185d4c8484308e45c3301e34" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 40381 } }
I20260620 18:32:54.828085 18394 sys_catalog.cc:564] T 00000000000000000000000000000000 P 012beec4185d4c8484308e45c3301e34 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:32:54.830693 18400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 012beec4185d4c8484308e45c3301e34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "012beec4185d4c8484308e45c3301e34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012beec4185d4c8484308e45c3301e34" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 40381 } } }
I20260620 18:32:54.831152 18400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 012beec4185d4c8484308e45c3301e34 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:54.830847 18401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 012beec4185d4c8484308e45c3301e34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 012beec4185d4c8484308e45c3301e34. Latest consensus state: current_term: 1 leader_uuid: "012beec4185d4c8484308e45c3301e34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012beec4185d4c8484308e45c3301e34" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 40381 } } }
I20260620 18:32:54.831570 18401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 012beec4185d4c8484308e45c3301e34 [sys.catalog]: This master's current role is: LEADER
I20260620 18:32:54.832995 18408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:32:54.837827 18408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:32:54.845494 18408 catalog_manager.cc:1347] Generated new cluster ID: 1c1c622400e4406585134eea1cee8718
I20260620 18:32:54.845597 18408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:32:54.862213 18408 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:32:54.864130 18408 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:32:54.872462 18408 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 012beec4185d4c8484308e45c3301e34: Generated new TSK 0
I20260620 18:32:54.873456 18408 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.10.131.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.131.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.131.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.1.keytab.
Entry for principal oryx/127.10.131.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.1.keytab.
I20260620 18:32:54.921880 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.1:0
--local_ip_for_outbound_sockets=127.10.131.1
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:40381
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.1.keytab
--principal=oryx/127.10.131.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:46739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:55.246065 18420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:55.246346 18420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:55.246505 18420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:55.253965 18420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:55.254072 18420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:55.254222 18420 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:32:55.254347 18420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.1
I20260620 18:32:55.264139 18420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:46739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.1.keytab
--principal=oryx/127.10.131.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.131.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.131.62:40381
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.1
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:55.265467 18420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:55.266782 18420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:55.279609 18425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:55.279551 18426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:55.279592 18428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:55.280795 18420 server_base.cc:1034] running on GCE node
I20260620 18:32:55.281490 18420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:55.282382 18420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:55.283627 18420 hybrid_clock.cc:648] HybridClock initialized: now 1781980375283565 us; error 56 us; skew 500 ppm
I20260620 18:32:55.283915 18420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:55 dist-test-slave-cbn3 krb5kdc[18304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980375, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:55.286913 18420 init.cc:377] Logged in from keytab as oryx/127.10.131.1@KRBTEST.COM (short username oryx)
I20260620 18:32:55.288166 18420 webserver.cc:460] Webserver started at http://127.10.131.1:41765/ using document root <none> and password file <none>
I20260620 18:32:55.288827 18420 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:55.288998 18420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:55.289337 18420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:55.292050 18420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-0/data/instance:
uuid: "7201b2b1dba94a56929d3723f0acd0a5"
format_stamp: "Formatted at 2026-06-20 18:32:55 on dist-test-slave-cbn3"
server_key: "95c0bad6b4db4432e6608d8486d7200c"
server_key_iv: "93ed180aac14f4668aaf3861e96f892c"
server_key_version: "encryptionkey@0"
I20260620 18:32:55.292804 18420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance:
uuid: "7201b2b1dba94a56929d3723f0acd0a5"
format_stamp: "Formatted at 2026-06-20 18:32:55 on dist-test-slave-cbn3"
server_key: "95c0bad6b4db4432e6608d8486d7200c"
server_key_iv: "93ed180aac14f4668aaf3861e96f892c"
server_key_version: "encryptionkey@0"
I20260620 18:32:55.297749 18420 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:32:55.301126 18435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:55.302398 18420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:32:55.302563 18420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-0/wal
uuid: "7201b2b1dba94a56929d3723f0acd0a5"
format_stamp: "Formatted at 2026-06-20 18:32:55 on dist-test-slave-cbn3"
server_key: "95c0bad6b4db4432e6608d8486d7200c"
server_key_iv: "93ed180aac14f4668aaf3861e96f892c"
server_key_version: "encryptionkey@0"
I20260620 18:32:55.302781 18420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:55.328303 18420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:55.332422 18420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:55.332721 18420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:55.335273 18420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:55.335383 18420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:55.335543 18420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:55.335616 18420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:32:55 dist-test-slave-cbn3 krb5kdc[18304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980375, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM
I20260620 18:32:55.360996 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:55.335930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:38525 (local address 127.10.131.62:40381)
0620 18:32:55.336553 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:55.336559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:55.337640 (+ 1081us) server_negotiation.cc:407] Connection header received
0620 18:32:55.340600 (+ 2960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:55.340608 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:55.340721 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:55.340825 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:55.343553 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:55.344458 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:55.347796 (+ 3338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:55.348242 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:55.348400 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:55.353493 (+ 5093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:55.353498 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:55.353501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:55.353531 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:55.357489 (+ 3958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:55.358389 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:55.358396 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:55.358400 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:55.358553 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:55.359241 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:55.359248 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:55.359253 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:55.359556 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:55.359713 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:55.360375 (+ 662us) server_negotiation.cc:299] Negotiation successful
0620 18:32:55.360585 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":355,"threads_started":1}
I20260620 18:32:55.362109 18445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:55.336584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:40381 (local address 127.10.131.1:38525)
0620 18:32:55.337467 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:55.337519 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:55.340242 (+ 2723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:55.341470 (+ 1228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:55.341484 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:55.342384 (+ 900us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:55.343356 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:55.343384 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:55.344650 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:55.344658 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:55.347599 (+ 2941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:55.347618 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:55.348482 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:55.348490 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:55.348782 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:55.349575 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:55.349606 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:55.353211 (+ 3605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:55.357629 (+ 4418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:55.357635 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:55.357649 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:55.358253 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:55.358689 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:55.358697 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:55.358715 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:55.359084 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:55.359685 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:55.359704 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:55.360036 (+ 332us) client_negotiation.cc:769] Sending connection context
0620 18:32:55.360284 (+ 248us) client_negotiation.cc:240] Negotiation successful
0620 18:32:55.360772 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":368,"threads_started":1}
I20260620 18:32:55.383293 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:55.367980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:44659 (local address 127.10.131.62:40381)
0620 18:32:55.368136 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:55.368141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:55.368766 (+ 625us) server_negotiation.cc:407] Connection header received
0620 18:32:55.368978 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:55.368982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:55.369044 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:55.369147 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:55.370933 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:55.371938 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:55.374347 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:55.374695 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:55.374784 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:55.377337 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:55.377341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:55.377344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:55.377371 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:55.380353 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:55.381287 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:55.381295 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:55.381301 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:55.381468 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:55.381880 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:55.381888 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:55.381892 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:55.382230 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:55.382346 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:55.382829 (+ 483us) server_negotiation.cc:299] Negotiation successful
0620 18:32:55.382977 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 18:32:55.383739 18508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:55.368125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:40381 (local address 127.10.131.1:44659)
0620 18:32:55.368676 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:55.368691 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:55.368835 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:55.369297 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:55.369307 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:55.370140 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:55.370762 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:55.370783 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:55.372102 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:55.372110 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:55.374202 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:55.374218 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:55.374910 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:55.374919 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:55.375171 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:55.375827 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:55.375857 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:55.377164 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:55.380481 (+ 3317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:55.380486 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:55.380493 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:55.381099 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:55.381564 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:55.381574 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:55.381577 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:55.381707 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:55.382346 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:55.382351 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:55.382570 (+ 219us) client_negotiation.cc:769] Sending connection context
0620 18:32:55.382745 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 18:32:55.383091 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":317,"threads_started":1}
I20260620 18:32:55.392616 18420 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.1:35645
I20260620 18:32:55.392652 18555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.1:35645 every 8 connection(s)
I20260620 18:32:55.393790 18420 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
I20260620 18:32:55.395666 18556 heartbeater.cc:347] Connected to a master server at 127.10.131.62:40381
I20260620 18:32:55.395897 18556 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:55.396474 18556 heartbeater.cc:511] Master 127.10.131.62:40381 requested a full tablet report, sending...
I20260620 18:32:55.398389 18349 ts_manager.cc:194] Registered new tserver with Master: 7201b2b1dba94a56929d3723f0acd0a5 (127.10.131.1:35645)
I20260620 18:32:55.399776 18349 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.131.1@KRBTEST.COM'} at 127.10.131.1:38525
I20260620 18:32:55.402755 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 18420
I20260620 18:32:55.402926 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance
I20260620 18:32:55.403545 10764 external_mini_cluster.cc:1428] Setting key bfea90fc9ef16e18cc4aa7aeacfd0a26
WARNING: no policy specified for oryx/127.10.131.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.131.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.131.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.2.keytab.
Entry for principal oryx/127.10.131.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.2.keytab.
I20260620 18:32:55.449513 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.2:0
--local_ip_for_outbound_sockets=127.10.131.2
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:40381
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.2.keytab
--principal=oryx/127.10.131.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:46739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:55.771303 18560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:55.771595 18560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:55.771692 18560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:55.778918 18560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:55.779047 18560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:55.779132 18560 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:32:55.779217 18560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.2
I20260620 18:32:55.791201 18560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:46739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.2.keytab
--principal=oryx/127.10.131.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.131.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.131.62:40381
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.2
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:55.792645 18560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:55.794377 18560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:55.808741 18568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:55.808775 18565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:55.808657 18566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:55.809715 18560 server_base.cc:1034] running on GCE node
I20260620 18:32:55.810289 18560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:55.811156 18560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:55.812359 18560 hybrid_clock.cc:648] HybridClock initialized: now 1781980375812291 us; error 48 us; skew 500 ppm
I20260620 18:32:55.812647 18560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:55 dist-test-slave-cbn3 krb5kdc[18304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980375, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:55.815618 18560 init.cc:377] Logged in from keytab as oryx/127.10.131.2@KRBTEST.COM (short username oryx)
I20260620 18:32:55.817075 18560 webserver.cc:460] Webserver started at http://127.10.131.2:43269/ using document root <none> and password file <none>
I20260620 18:32:55.817812 18560 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:55.817946 18560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:55.818320 18560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:55.821360 18560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-1/data/instance:
uuid: "74b98e19928d4abdbd1e062eddc64d68"
format_stamp: "Formatted at 2026-06-20 18:32:55 on dist-test-slave-cbn3"
server_key: "dc24566a7623c8ccd7e4a4da40b63bb0"
server_key_iv: "08910f1c2cb197ab937f3c36b2120f71"
server_key_version: "encryptionkey@0"
I20260620 18:32:55.822157 18560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance:
uuid: "74b98e19928d4abdbd1e062eddc64d68"
format_stamp: "Formatted at 2026-06-20 18:32:55 on dist-test-slave-cbn3"
server_key: "dc24566a7623c8ccd7e4a4da40b63bb0"
server_key_iv: "08910f1c2cb197ab937f3c36b2120f71"
server_key_version: "encryptionkey@0"
I20260620 18:32:55.827065 18560 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:32:55.830659 18575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:55.831982 18560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:32:55.832134 18560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-1/wal
uuid: "74b98e19928d4abdbd1e062eddc64d68"
format_stamp: "Formatted at 2026-06-20 18:32:55 on dist-test-slave-cbn3"
server_key: "dc24566a7623c8ccd7e4a4da40b63bb0"
server_key_iv: "08910f1c2cb197ab937f3c36b2120f71"
server_key_version: "encryptionkey@0"
I20260620 18:32:55.832358 18560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:55.855360 18560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:55.859210 18560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:55.859503 18560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:55.862094 18560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:55.862183 18560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:55.862442 18560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:55.862572 18560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:32:55 dist-test-slave-cbn3 krb5kdc[18304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980375, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.2@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM
I20260620 18:32:55.885151 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:55.862520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:44403 (local address 127.10.131.62:40381)
0620 18:32:55.862722 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:55.862730 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:55.864172 (+ 1442us) server_negotiation.cc:407] Connection header received
0620 18:32:55.866241 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:55.866247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:55.866330 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:55.866447 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:55.868522 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:55.869953 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:55.872367 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:55.873065 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:55.873204 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:55.877675 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:55.877680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:55.877683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:55.877715 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:55.881483 (+ 3768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:55.882629 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:55.882635 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:55.882641 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:55.882819 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:55.883342 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:55.883350 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:55.883354 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:55.883622 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:55.883767 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:55.884548 (+ 781us) server_negotiation.cc:299] Negotiation successful
0620 18:32:55.884760 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 18:32:55.886175 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:55.863056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:40381 (local address 127.10.131.2:44403)
0620 18:32:55.863998 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:55.864041 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:55.865909 (+ 1868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:55.866666 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:55.866674 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:55.867460 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:55.868277 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:55.868316 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:55.870128 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:55.870135 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:55.872201 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:55.872215 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:55.873301 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:55.873309 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:55.873566 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:55.874360 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:55.874405 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:55.877379 (+ 2974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:55.881643 (+ 4264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:55.881655 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:55.881674 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:55.882425 (+ 751us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:55.882957 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:55.882964 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:55.882969 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:55.883217 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:55.883753 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:55.883761 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:55.884132 (+ 371us) client_negotiation.cc:769] Sending connection context
0620 18:32:55.884450 (+ 318us) client_negotiation.cc:240] Negotiation successful
0620 18:32:55.884975 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":408,"threads_started":1}
I20260620 18:32:55.908200 18643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:55.891624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:40381 (local address 127.10.131.2:58385)
0620 18:32:55.892325 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:55.892343 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:55.892502 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:55.892909 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:55.892918 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:55.893804 (+ 886us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:55.894344 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:55.894362 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:55.895972 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:55.895980 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:55.897272 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:55.897290 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:55.898697 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:55.898707 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:55.899013 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:55.899746 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:55.899775 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:55.901188 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:55.904977 (+ 3789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:55.904982 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:55.904986 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:55.905730 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:55.906254 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:55.906261 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:55.906267 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:55.906436 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:55.907209 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:55.907215 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:55.907453 (+ 238us) client_negotiation.cc:769] Sending connection context
0620 18:32:55.907642 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 18:32:55.907878 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":463,"threads_started":1}
I20260620 18:32:55.908349 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:55.891507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:58385 (local address 127.10.131.62:40381)
0620 18:32:55.891651 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:55.891655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:55.892429 (+ 774us) server_negotiation.cc:407] Connection header received
0620 18:32:55.892632 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:55.892638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:55.892734 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:55.892856 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:55.894547 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:55.895822 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:55.897504 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:55.898266 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:55.898403 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:55.901350 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:55.901355 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:55.901358 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:55.901387 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:55.904818 (+ 3431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:55.905883 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:55.905900 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:55.905906 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:55.906076 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:55.906572 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:55.906577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:55.906581 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:55.907055 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:55.907224 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:55.907757 (+ 533us) server_negotiation.cc:299] Negotiation successful
0620 18:32:55.907974 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 18:32:55.909765 18560 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.2:41595
I20260620 18:32:55.909718 18694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.2:41595 every 8 connection(s)
I20260620 18:32:55.911316 18560 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
I20260620 18:32:55.913427 18695 heartbeater.cc:347] Connected to a master server at 127.10.131.62:40381
I20260620 18:32:55.913643 18695 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:55.914212 18695 heartbeater.cc:511] Master 127.10.131.62:40381 requested a full tablet report, sending...
I20260620 18:32:55.915406 18349 ts_manager.cc:194] Registered new tserver with Master: 74b98e19928d4abdbd1e062eddc64d68 (127.10.131.2:41595)
I20260620 18:32:55.916280 18349 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.131.2@KRBTEST.COM'} at 127.10.131.2:44403
I20260620 18:32:55.920483 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 18560
I20260620 18:32:55.920645 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance
I20260620 18:32:55.921152 10764 external_mini_cluster.cc:1428] Setting key f60e7c405c09e2e6fdce8ef06a9c119a
WARNING: no policy specified for oryx/127.10.131.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.131.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.131.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.3.keytab.
Entry for principal oryx/127.10.131.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.3.keytab.
I20260620 18:32:55.963836 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.3:0
--local_ip_for_outbound_sockets=127.10.131.3
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:40381
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.3.keytab
--principal=oryx/127.10.131.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:46739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:32:56.277005 18699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:32:56.277336 18699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:32:56.277510 18699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:32:56.284541 18699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:32:56.284674 18699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:32:56.284821 18699 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:32:56.284973 18699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.3
I20260620 18:32:56.294317 18699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:46739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.3.keytab
--principal=oryx/127.10.131.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.131.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.131.62:40381
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.3
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:32:56.295531 18699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:32:56.296831 18699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:32:56.309810 18704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:56.309926 18705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:32:56.309897 18707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:32:56.310667 18699 server_base.cc:1034] running on GCE node
I20260620 18:32:56.311255 18699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:32:56.312122 18699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:32:56.313357 18699 hybrid_clock.cc:648] HybridClock initialized: now 1781980376313283 us; error 49 us; skew 500 ppm
I20260620 18:32:56.313658 18699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:32:56 dist-test-slave-cbn3 krb5kdc[18304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980376, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:32:56.316665 18699 init.cc:377] Logged in from keytab as oryx/127.10.131.3@KRBTEST.COM (short username oryx)
I20260620 18:32:56.318094 18699 webserver.cc:460] Webserver started at http://127.10.131.3:43855/ using document root <none> and password file <none>
I20260620 18:32:56.318742 18699 fs_manager.cc:359] Metadata directory not provided
I20260620 18:32:56.318872 18699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:32:56.319188 18699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:32:56.322000 18699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-2/data/instance:
uuid: "58f75de98c4c4930b6ee00f9c2c1a33f"
format_stamp: "Formatted at 2026-06-20 18:32:56 on dist-test-slave-cbn3"
server_key: "30da7f1b2059b83294a97aa38a64635d"
server_key_iv: "3d4ea88cbccfb3210c9c09a59ee16263"
server_key_version: "encryptionkey@0"
I20260620 18:32:56.322726 18699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance:
uuid: "58f75de98c4c4930b6ee00f9c2c1a33f"
format_stamp: "Formatted at 2026-06-20 18:32:56 on dist-test-slave-cbn3"
server_key: "30da7f1b2059b83294a97aa38a64635d"
server_key_iv: "3d4ea88cbccfb3210c9c09a59ee16263"
server_key_version: "encryptionkey@0"
I20260620 18:32:56.327596 18699 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 18:32:56.331017 18714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:56.332237 18699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 18:32:56.332402 18699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-2/wal
uuid: "58f75de98c4c4930b6ee00f9c2c1a33f"
format_stamp: "Formatted at 2026-06-20 18:32:56 on dist-test-slave-cbn3"
server_key: "30da7f1b2059b83294a97aa38a64635d"
server_key_iv: "3d4ea88cbccfb3210c9c09a59ee16263"
server_key_version: "encryptionkey@0"
I20260620 18:32:56.332602 18699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:32:56.366773 18699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:32:56.370947 18699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:32:56.371202 18699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:32:56.373921 18699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:32:56.374028 18699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:32:56.374199 18699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:32:56.374325 18699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 20 18:32:56 dist-test-slave-cbn3 krb5kdc[18304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980376, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.3@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM
I20260620 18:32:56.396978 18723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:56.374802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:40381 (local address 127.10.131.3:42571)
0620 18:32:56.375629 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:56.375681 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:56.377522 (+ 1841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:56.378216 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:56.378224 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:56.378932 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:56.379750 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:56.379791 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:56.381484 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:56.381490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:56.383644 (+ 2154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:56.383655 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:56.384456 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:56.384464 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:56.384665 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:56.385329 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:56.385354 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:56.388158 (+ 2804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:56.392343 (+ 4185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:56.392352 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:56.392366 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:56.392921 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:56.393406 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:56.393412 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:56.393415 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:56.393654 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:56.394363 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:56.394373 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:56.394778 (+ 405us) client_negotiation.cc:769] Sending connection context
0620 18:32:56.395042 (+ 264us) client_negotiation.cc:240] Negotiation successful
0620 18:32:56.395515 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":365,"threads_started":1}
I20260620 18:32:56.397796 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:56.374351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:42571 (local address 127.10.131.62:40381)
0620 18:32:56.374562 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:56.374569 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:56.375808 (+ 1239us) server_negotiation.cc:407] Connection header received
0620 18:32:56.377789 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:56.377795 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:56.377885 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:56.378001 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:56.379972 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:56.381276 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:56.383823 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:56.384214 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:56.384342 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:56.388367 (+ 4025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:56.388372 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:56.388375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:56.388406 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:56.392180 (+ 3774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:56.393080 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:56.393087 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:56.393094 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:56.393275 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:56.393776 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:56.393781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:56.393785 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:56.394127 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:56.394296 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:56.396971 (+ 2675us) server_negotiation.cc:299] Negotiation successful
0620 18:32:56.397303 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:32:56.402422 18556 heartbeater.cc:503] Master 127.10.131.62:40381 was elected leader, sending a full tablet report...
I20260620 18:32:56.424104 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:56.406097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:48643 (local address 127.10.131.62:40381)
0620 18:32:56.406274 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:56.406280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:56.409578 (+ 3298us) server_negotiation.cc:407] Connection header received
0620 18:32:56.409779 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:56.409786 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:56.409888 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:56.410008 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:56.411442 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:56.412147 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:56.414623 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:56.415021 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:56.415110 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:56.417934 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:56.417941 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:56.417946 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:56.417997 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:56.421377 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:56.422194 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:56.422198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:56.422200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:56.422330 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:56.422770 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:56.422775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:56.422777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:56.423057 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:56.423175 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:56.423707 (+ 532us) server_negotiation.cc:299] Negotiation successful
0620 18:32:56.423856 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:32:56.424765 18790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:56.406168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:40381 (local address 127.10.131.3:48643)
0620 18:32:56.409475 (+ 3307us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:56.409496 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:56.409643 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:56.410063 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:56.410072 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:56.410802 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:56.411267 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:56.411282 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:56.412292 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:56.412301 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:56.414471 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:56.414491 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:56.415210 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:56.415220 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:56.415468 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:56.416191 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:56.416222 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:56.417653 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:56.421501 (+ 3848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:56.421506 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:56.421510 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:56.422077 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:56.422433 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:56.422441 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:56.422447 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:56.422637 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:56.423181 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:56.423189 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:56.423447 (+ 258us) client_negotiation.cc:769] Sending connection context
0620 18:32:56.423620 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 18:32:56.424022 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3143,"thread_start_us":2919,"threads_started":1}
I20260620 18:32:56.435096 18699 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.3:42999
I20260620 18:32:56.435173 18833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.3:42999 every 8 connection(s)
I20260620 18:32:56.436513 18699 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
I20260620 18:32:56.438911 18834 heartbeater.cc:347] Connected to a master server at 127.10.131.62:40381
I20260620 18:32:56.439196 18834 heartbeater.cc:464] Registering TS with master...
I20260620 18:32:56.439828 18834 heartbeater.cc:511] Master 127.10.131.62:40381 requested a full tablet report, sending...
I20260620 18:32:56.441236 18349 ts_manager.cc:194] Registered new tserver with Master: 58f75de98c4c4930b6ee00f9c2c1a33f (127.10.131.3:42999)
I20260620 18:32:56.442052 18349 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.131.3@KRBTEST.COM'} at 127.10.131.3:42571
I20260620 18:32:56.443167 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 18699
I20260620 18:32:56.443338 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance
I20260620 18:32:56.443872 10764 external_mini_cluster.cc:1428] Setting key 1af055310a739218be835089a04e4977
I20260620 18:32:56.446695 10764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 18:32:56 dist-test-slave-cbn3 krb5kdc[18304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980376, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 18:32:56 dist-test-slave-cbn3 krb5kdc[18304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:32:56 dist-test-slave-cbn3 krb5kdc[18304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:32:56.918601 18695 heartbeater.cc:503] Master 127.10.131.62:40381 was elected leader, sending a full tablet report...
I20260620 18:32:57.444938 18834 heartbeater.cc:503] Master 127.10.131.62:40381 was elected leader, sending a full tablet report...
W20260620 18:32:57.533552 18842 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.044s user 0.007s sys 0.007s
W20260620 18:32:57.534698 18444 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:32:56.484173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50930 (local address 127.10.131.62:40381)
0620 18:32:56.484341 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:56.484346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:56.484595 (+ 249us) server_negotiation.cc:407] Connection header received
0620 18:32:56.484741 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:56.484745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:56.484803 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:56.484957 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:56.486170 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:56.486881 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:56.488110 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:56.488580 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:56.488690 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:57.534455 (+1045765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50930: BlockingRecv error: recv got EOF from 127.0.0.1:50930 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 18:32:57 dist-test-slave-cbn3 krb5kdc[18304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980376, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM
I20260620 18:32:57.556293 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:57.539221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50944 (local address 127.10.131.62:40381)
0620 18:32:57.539409 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:57.539419 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:57.539749 (+ 330us) server_negotiation.cc:407] Connection header received
0620 18:32:57.539960 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:57.539965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:57.540055 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:57.540160 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:57.541443 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:57.542178 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:57.543527 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:57.543944 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:57.544096 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:57.551844 (+ 7748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:57.551848 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:57.551852 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:57.551884 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:57.554295 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:57.554860 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:57.554864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:57.554866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:57.554968 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:57.555248 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:57.555251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:57.555253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:57.555501 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:57.555687 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:57.555896 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 18:32:57.556059 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 18:32:57.560204 18349 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:32:57.566938 18349 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:32:57.588887 18858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:57.580895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44582 (local address 127.10.131.1:35645)
0620 18:32:57.581820 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:57.581827 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:57.581853 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:32:57.581939 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:57.581947 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:57.582149 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:57.582919 (+ 770us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:57.583057 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:57.584150 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:57.586644 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:57.587760 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:57.587905 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:57.588124 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:57.588389 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 18:32:57.588547 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":793,"thread_start_us":627,"threads_started":1}
I20260620 18:32:57.589087 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:57.580354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:35645 (local address 127.0.0.1:44582)
0620 18:32:57.581130 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:57.581147 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:57.581284 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:57.582346 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:57.582350 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:57.582371 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:57.582767 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:57.582777 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:57.584312 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:57.584316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:57.586501 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:57.586509 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:57.587964 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:57.587974 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:57.588178 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:57.588220 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 18:32:57.588352 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 18:32:57.588567 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"mutex_wait_us":48,"thread_start_us":693,"threads_started":1}
I20260620 18:32:57.589254 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:57.578449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56828 (local address 127.10.131.3:42999)
0620 18:32:57.580773 (+ 2324us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:57.580781 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:57.580807 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:32:57.580929 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:57.580940 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:57.581166 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:57.582027 (+ 861us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:57.582252 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:57.583651 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:57.586413 (+ 2762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:57.587787 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:57.588551 (+ 764us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:57.588738 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:57.588796 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 18:32:57.588974 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2178,"thread_start_us":2051,"threads_started":1}
I20260620 18:32:57.589190 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:57.578771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:41595 (local address 127.0.0.1:42922)
0620 18:32:57.580201 (+ 1430us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:57.580218 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:57.580322 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:57.581001 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:57.581010 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:57.581057 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:57.581556 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:57.581587 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:57.583714 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:57.583720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:57.586097 (+ 2377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:57.586109 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:57.588337 (+ 2228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:57.588345 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:57.588538 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:57.588566 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 18:32:57.588655 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 18:32:57.588770 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1275,"mutex_wait_us":188,"thread_start_us":409,"threads_started":1}
I20260620 18:32:57.589828 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:57.578228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:42999 (local address 127.0.0.1:56828)
0620 18:32:57.579552 (+ 1324us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:57.579603 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:57.579756 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:57.581410 (+ 1654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:57.581418 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:57.581442 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:57.581865 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:57.581876 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:57.583828 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:57.583834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:57.586266 (+ 2432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:57.586279 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:57.588057 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:57.588063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:57.588222 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:57.588259 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 18:32:57.588337 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 18:32:57.589455 (+ 1118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1091,"mutex_wait_us":948,"thread_start_us":1660,"threads_started":1}
I20260620 18:32:57.590284 18856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:57.579034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42922 (local address 127.10.131.2:41595)
0620 18:32:57.580310 (+ 1276us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:57.580319 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:57.580347 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:32:57.580455 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:57.580465 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:57.580723 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:57.581828 (+ 1105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:57.582026 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:57.583563 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:57.586271 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:57.587787 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:57.588039 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:57.588229 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:57.589822 (+ 1593us) server_negotiation.cc:299] Negotiation successful
0620 18:32:57.589983 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1120,"thread_start_us":977,"threads_started":1}
I20260620 18:32:57.593001 18625 tablet_service.cc:1467] Processing CreateTablet for tablet 57ddc99075d1455fbf3bf52d35641a04 (DEFAULT_TABLE table=test-table [id=5755b67a74d24a8b9b8b10ea1a38274f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:57.593012 18486 tablet_service.cc:1467] Processing CreateTablet for tablet 57ddc99075d1455fbf3bf52d35641a04 (DEFAULT_TABLE table=test-table [id=5755b67a74d24a8b9b8b10ea1a38274f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:57.593432 18764 tablet_service.cc:1467] Processing CreateTablet for tablet 57ddc99075d1455fbf3bf52d35641a04 (DEFAULT_TABLE table=test-table [id=5755b67a74d24a8b9b8b10ea1a38274f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:57.594233 18625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57ddc99075d1455fbf3bf52d35641a04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:57.594285 18486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57ddc99075d1455fbf3bf52d35641a04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:57.594555 18764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57ddc99075d1455fbf3bf52d35641a04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:57.604827 18859 tablet_bootstrap.cc:492] T 57ddc99075d1455fbf3bf52d35641a04 P 74b98e19928d4abdbd1e062eddc64d68: Bootstrap starting.
I20260620 18:32:57.608421 18859 tablet_bootstrap.cc:654] T 57ddc99075d1455fbf3bf52d35641a04 P 74b98e19928d4abdbd1e062eddc64d68: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:57.609439 18861 tablet_bootstrap.cc:492] T 57ddc99075d1455fbf3bf52d35641a04 P 58f75de98c4c4930b6ee00f9c2c1a33f: Bootstrap starting.
I20260620 18:32:57.610420 18859 log.cc:824] T 57ddc99075d1455fbf3bf52d35641a04 P 74b98e19928d4abdbd1e062eddc64d68: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:57.612978 18860 tablet_bootstrap.cc:492] T 57ddc99075d1455fbf3bf52d35641a04 P 7201b2b1dba94a56929d3723f0acd0a5: Bootstrap starting.
I20260620 18:32:57.615339 18861 tablet_bootstrap.cc:654] T 57ddc99075d1455fbf3bf52d35641a04 P 58f75de98c4c4930b6ee00f9c2c1a33f: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:57.616072 18859 tablet_bootstrap.cc:492] T 57ddc99075d1455fbf3bf52d35641a04 P 74b98e19928d4abdbd1e062eddc64d68: No bootstrap required, opened a new log
I20260620 18:32:57.616410 18859 ts_tablet_manager.cc:1397] T 57ddc99075d1455fbf3bf52d35641a04 P 74b98e19928d4abdbd1e062eddc64d68: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20260620 18:32:57.616721 18861 log.cc:824] T 57ddc99075d1455fbf3bf52d35641a04 P 58f75de98c4c4930b6ee00f9c2c1a33f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:57.617882 18860 tablet_bootstrap.cc:654] T 57ddc99075d1455fbf3bf52d35641a04 P 7201b2b1dba94a56929d3723f0acd0a5: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:57.619405 18860 log.cc:824] T 57ddc99075d1455fbf3bf52d35641a04 P 7201b2b1dba94a56929d3723f0acd0a5: Log is configured to *not* fsync() on all Append() calls
I20260620 18:32:57.623817 18859 raft_consensus.cc:348] T 57ddc99075d1455fbf3bf52d35641a04 P 74b98e19928d4abdbd1e062eddc64d68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } } peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } }
I20260620 18:32:57.624279 18859 raft_consensus.cc:374] T 57ddc99075d1455fbf3bf52d35641a04 P 74b98e19928d4abdbd1e062eddc64d68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:57.624418 18859 raft_consensus.cc:729] T 57ddc99075d1455fbf3bf52d35641a04 P 74b98e19928d4abdbd1e062eddc64d68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74b98e19928d4abdbd1e062eddc64d68, State: Initialized, Role: FOLLOWER
I20260620 18:32:57.625092 18859 consensus_queue.cc:260] T 57ddc99075d1455fbf3bf52d35641a04 P 74b98e19928d4abdbd1e062eddc64d68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } } peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } }
I20260620 18:32:57.625982 18861 tablet_bootstrap.cc:492] T 57ddc99075d1455fbf3bf52d35641a04 P 58f75de98c4c4930b6ee00f9c2c1a33f: No bootstrap required, opened a new log
I20260620 18:32:57.626107 18860 tablet_bootstrap.cc:492] T 57ddc99075d1455fbf3bf52d35641a04 P 7201b2b1dba94a56929d3723f0acd0a5: No bootstrap required, opened a new log
I20260620 18:32:57.626312 18861 ts_tablet_manager.cc:1397] T 57ddc99075d1455fbf3bf52d35641a04 P 58f75de98c4c4930b6ee00f9c2c1a33f: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.012s
I20260620 18:32:57.626420 18860 ts_tablet_manager.cc:1397] T 57ddc99075d1455fbf3bf52d35641a04 P 7201b2b1dba94a56929d3723f0acd0a5: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260620 18:32:57.626842 18859 ts_tablet_manager.cc:1428] T 57ddc99075d1455fbf3bf52d35641a04 P 74b98e19928d4abdbd1e062eddc64d68: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260620 18:32:57.636991 18860 raft_consensus.cc:348] T 57ddc99075d1455fbf3bf52d35641a04 P 7201b2b1dba94a56929d3723f0acd0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } } peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } }
I20260620 18:32:57.637441 18860 raft_consensus.cc:374] T 57ddc99075d1455fbf3bf52d35641a04 P 7201b2b1dba94a56929d3723f0acd0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:57.637537 18860 raft_consensus.cc:729] T 57ddc99075d1455fbf3bf52d35641a04 P 7201b2b1dba94a56929d3723f0acd0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7201b2b1dba94a56929d3723f0acd0a5, State: Initialized, Role: FOLLOWER
I20260620 18:32:57.637419 18861 raft_consensus.cc:348] T 57ddc99075d1455fbf3bf52d35641a04 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } } peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } }
I20260620 18:32:57.637820 18861 raft_consensus.cc:374] T 57ddc99075d1455fbf3bf52d35641a04 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:57.637917 18861 raft_consensus.cc:729] T 57ddc99075d1455fbf3bf52d35641a04 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f75de98c4c4930b6ee00f9c2c1a33f, State: Initialized, Role: FOLLOWER
I20260620 18:32:57.638177 18860 consensus_queue.cc:260] T 57ddc99075d1455fbf3bf52d35641a04 P 7201b2b1dba94a56929d3723f0acd0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } } peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } }
I20260620 18:32:57.638471 18861 consensus_queue.cc:260] T 57ddc99075d1455fbf3bf52d35641a04 P 58f75de98c4c4930b6ee00f9c2c1a33f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } } peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } }
I20260620 18:32:57.639961 18860 ts_tablet_manager.cc:1428] T 57ddc99075d1455fbf3bf52d35641a04 P 7201b2b1dba94a56929d3723f0acd0a5: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260620 18:32:57.640051 18861 ts_tablet_manager.cc:1428] T 57ddc99075d1455fbf3bf52d35641a04 P 58f75de98c4c4930b6ee00f9c2c1a33f: Time spent starting tablet: real 0.014s user 0.011s sys 0.001s
W20260620 18:32:57.646695 18557 tablet.cc:2369] T 57ddc99075d1455fbf3bf52d35641a04 P 7201b2b1dba94a56929d3723f0acd0a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:32:57.664510 18696 tablet.cc:2369] T 57ddc99075d1455fbf3bf52d35641a04 P 74b98e19928d4abdbd1e062eddc64d68: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:32:57.685452 18866 raft_consensus.cc:482] T 57ddc99075d1455fbf3bf52d35641a04 P 7201b2b1dba94a56929d3723f0acd0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:32:57.685746 18866 raft_consensus.cc:504] T 57ddc99075d1455fbf3bf52d35641a04 P 7201b2b1dba94a56929d3723f0acd0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } } peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } }
I20260620 18:32:57.687379 18866 leader_election.cc:288] T 57ddc99075d1455fbf3bf52d35641a04 P 7201b2b1dba94a56929d3723f0acd0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58f75de98c4c4930b6ee00f9c2c1a33f (127.10.131.3:42999), 74b98e19928d4abdbd1e062eddc64d68 (127.10.131.2:41595)
W20260620 18:32:57.688809 18835 tablet.cc:2369] T 57ddc99075d1455fbf3bf52d35641a04 P 58f75de98c4c4930b6ee00f9c2c1a33f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:32:57.693758 18856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:57.687914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:43267 (local address 127.10.131.2:41595)
0620 18:32:57.688138 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:57.688145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:57.688652 (+ 507us) server_negotiation.cc:407] Connection header received
0620 18:32:57.688949 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:57.688956 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:57.689033 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:57.689115 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:57.689642 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:57.690322 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:57.692107 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:57.692817 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:57.693049 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:57.693123 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:57.693390 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 18:32:57.693512 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:32:57.693861 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:57.687690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:55397 (local address 127.10.131.3:42999)
0620 18:32:57.687890 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:57.687896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:57.688806 (+ 910us) server_negotiation.cc:407] Connection header received
0620 18:32:57.689050 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:57.689055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:57.689125 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:57.689193 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:57.689947 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:57.690606 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:57.692215 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:57.693001 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:57.693185 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:57.693254 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:57.693518 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 18:32:57.693634 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:32:57.693905 18868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:57.687683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:41595 (local address 127.10.131.1:43267)
0620 18:32:57.688516 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:57.688536 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:57.688725 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:57.689153 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:57.689160 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:57.689182 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:57.689498 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:57.689506 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:57.690440 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:57.690448 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:57.691954 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:57.691965 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:57.693062 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:57.693067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:57.693225 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:57.693249 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 18:32:57.693366 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 18:32:57.693483 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"mutex_wait_us":4,"thread_start_us":457,"threads_started":1}
I20260620 18:32:57.693959 18869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:57.687791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:42999 (local address 127.10.131.1:55397)
0620 18:32:57.688650 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:57.688709 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:57.688864 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:57.689385 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:57.689392 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:57.689414 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:57.689802 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:57.689810 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:57.690740 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:57.690745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:57.692066 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:57.692077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:57.693207 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:57.693214 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:57.693372 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:57.693393 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 18:32:57.693495 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 18:32:57.693612 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":527,"threads_started":1}
I20260620 18:32:57.694710 18650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ddc99075d1455fbf3bf52d35641a04" candidate_uuid: "7201b2b1dba94a56929d3723f0acd0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74b98e19928d4abdbd1e062eddc64d68" is_pre_election: true
I20260620 18:32:57.694856 18785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ddc99075d1455fbf3bf52d35641a04" candidate_uuid: "7201b2b1dba94a56929d3723f0acd0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" is_pre_election: true
I20260620 18:32:57.695186 18650 raft_consensus.cc:2454] T 57ddc99075d1455fbf3bf52d35641a04 P 74b98e19928d4abdbd1e062eddc64d68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201b2b1dba94a56929d3723f0acd0a5 in term 0.
I20260620 18:32:57.695320 18785 raft_consensus.cc:2454] T 57ddc99075d1455fbf3bf52d35641a04 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201b2b1dba94a56929d3723f0acd0a5 in term 0.
I20260620 18:32:57.695842 18439 leader_election.cc:302] T 57ddc99075d1455fbf3bf52d35641a04 P 7201b2b1dba94a56929d3723f0acd0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7201b2b1dba94a56929d3723f0acd0a5, 74b98e19928d4abdbd1e062eddc64d68; no voters:
I20260620 18:32:57.696235 18866 raft_consensus.cc:2781] T 57ddc99075d1455fbf3bf52d35641a04 P 7201b2b1dba94a56929d3723f0acd0a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:32:57.696373 18866 raft_consensus.cc:482] T 57ddc99075d1455fbf3bf52d35641a04 P 7201b2b1dba94a56929d3723f0acd0a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:32:57.696478 18866 raft_consensus.cc:3037] T 57ddc99075d1455fbf3bf52d35641a04 P 7201b2b1dba94a56929d3723f0acd0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:57.698602 18866 raft_consensus.cc:504] T 57ddc99075d1455fbf3bf52d35641a04 P 7201b2b1dba94a56929d3723f0acd0a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } } peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } }
I20260620 18:32:57.699261 18866 leader_election.cc:288] T 57ddc99075d1455fbf3bf52d35641a04 P 7201b2b1dba94a56929d3723f0acd0a5 [CANDIDATE]: Term 1 election: Requested vote from peers 58f75de98c4c4930b6ee00f9c2c1a33f (127.10.131.3:42999), 74b98e19928d4abdbd1e062eddc64d68 (127.10.131.2:41595)
I20260620 18:32:57.699635 18785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ddc99075d1455fbf3bf52d35641a04" candidate_uuid: "7201b2b1dba94a56929d3723f0acd0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f"
I20260620 18:32:57.699733 18650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ddc99075d1455fbf3bf52d35641a04" candidate_uuid: "7201b2b1dba94a56929d3723f0acd0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74b98e19928d4abdbd1e062eddc64d68"
I20260620 18:32:57.699837 18785 raft_consensus.cc:3037] T 57ddc99075d1455fbf3bf52d35641a04 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:57.699944 18650 raft_consensus.cc:3037] T 57ddc99075d1455fbf3bf52d35641a04 P 74b98e19928d4abdbd1e062eddc64d68 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:57.702210 18650 raft_consensus.cc:2454] T 57ddc99075d1455fbf3bf52d35641a04 P 74b98e19928d4abdbd1e062eddc64d68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201b2b1dba94a56929d3723f0acd0a5 in term 1.
I20260620 18:32:57.702239 18785 raft_consensus.cc:2454] T 57ddc99075d1455fbf3bf52d35641a04 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201b2b1dba94a56929d3723f0acd0a5 in term 1.
I20260620 18:32:57.702620 18439 leader_election.cc:302] T 57ddc99075d1455fbf3bf52d35641a04 P 7201b2b1dba94a56929d3723f0acd0a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7201b2b1dba94a56929d3723f0acd0a5, 74b98e19928d4abdbd1e062eddc64d68; no voters:
I20260620 18:32:57.702893 18866 raft_consensus.cc:2781] T 57ddc99075d1455fbf3bf52d35641a04 P 7201b2b1dba94a56929d3723f0acd0a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:57.703317 18866 raft_consensus.cc:686] T 57ddc99075d1455fbf3bf52d35641a04 P 7201b2b1dba94a56929d3723f0acd0a5 [term 1 LEADER]: Becoming Leader. State: Replica: 7201b2b1dba94a56929d3723f0acd0a5, State: Running, Role: LEADER
I20260620 18:32:57.703797 18866 consensus_queue.cc:237] T 57ddc99075d1455fbf3bf52d35641a04 P 7201b2b1dba94a56929d3723f0acd0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } } peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } }
I20260620 18:32:57.708421 18348 catalog_manager.cc:5495] T 57ddc99075d1455fbf3bf52d35641a04 P 7201b2b1dba94a56929d3723f0acd0a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7201b2b1dba94a56929d3723f0acd0a5 (127.10.131.1). New cstate: current_term: 1 leader_uuid: "7201b2b1dba94a56929d3723f0acd0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } health_report { overall_health: UNKNOWN } } }
I20260620 18:32:57.741590 18858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:57.736557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44584 (local address 127.10.131.1:35645)
0620 18:32:57.736789 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:57.736795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:57.737014 (+ 219us) server_negotiation.cc:407] Connection header received
0620 18:32:57.737173 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:57.737176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:57.737251 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:57.737403 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:32:57.737657 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:57.738368 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:57.739624 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:57.740009 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:57.740105 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:57.740474 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:32:57.741044 (+ 570us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:32:57.741167 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:57.741239 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 18:32:57.741299 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 18:32:57.749315 18650 raft_consensus.cc:1261] T 57ddc99075d1455fbf3bf52d35641a04 P 74b98e19928d4abdbd1e062eddc64d68 [term 1 FOLLOWER]: Refusing update from remote peer 7201b2b1dba94a56929d3723f0acd0a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:57.749341 18785 raft_consensus.cc:1261] T 57ddc99075d1455fbf3bf52d35641a04 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 1 FOLLOWER]: Refusing update from remote peer 7201b2b1dba94a56929d3723f0acd0a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:57.750267 18870 consensus_queue.cc:1035] T 57ddc99075d1455fbf3bf52d35641a04 P 7201b2b1dba94a56929d3723f0acd0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:57.750597 18866 consensus_queue.cc:1035] T 57ddc99075d1455fbf3bf52d35641a04 P 7201b2b1dba94a56929d3723f0acd0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:57.761129 18873 mvcc.cc:205] Tried to move back new op lower bound from 7298991627251646464 to 7298991627077947392. Current Snapshot: MvccSnapshot[applied={T|T < 7298991627251646464}]
I20260620 18:32:57.763746 18875 mvcc.cc:205] Tried to move back new op lower bound from 7298991627251646464 to 7298991627077947392. Current Snapshot: MvccSnapshot[applied={T|T < 7298991627251646464}]
I20260620 18:32:57.777571 18348 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:32:57.777865 18348 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:32:57.781486 18348 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 012beec4185d4c8484308e45c3301e34: Sending DeleteTablet for 3 replicas of tablet 57ddc99075d1455fbf3bf52d35641a04
I20260620 18:32:57.782594 18764 tablet_service.cc:1514] Processing DeleteTablet for tablet 57ddc99075d1455fbf3bf52d35641a04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:57 UTC) from {username='oryx', principal='oryx/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:56828
I20260620 18:32:57.782742 18625 tablet_service.cc:1514] Processing DeleteTablet for tablet 57ddc99075d1455fbf3bf52d35641a04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:57 UTC) from {username='oryx', principal='oryx/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:42922
I20260620 18:32:57.783327 18486 tablet_service.cc:1514] Processing DeleteTablet for tablet 57ddc99075d1455fbf3bf52d35641a04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:57 UTC) from {username='oryx', principal='oryx/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:44582
I20260620 18:32:57.784102 18348 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:32:57.784968 18348 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:32:57.795138 18884 tablet_replica.cc:330] T 57ddc99075d1455fbf3bf52d35641a04 P 74b98e19928d4abdbd1e062eddc64d68: stopping tablet replica
I20260620 18:32:57.795853 18885 tablet_replica.cc:330] T 57ddc99075d1455fbf3bf52d35641a04 P 7201b2b1dba94a56929d3723f0acd0a5: stopping tablet replica
I20260620 18:32:57.796618 18885 raft_consensus.cc:2229] T 57ddc99075d1455fbf3bf52d35641a04 P 7201b2b1dba94a56929d3723f0acd0a5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:32:57.797278 18884 raft_consensus.cc:2229] T 57ddc99075d1455fbf3bf52d35641a04 P 74b98e19928d4abdbd1e062eddc64d68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:32:57.797937 18884 raft_consensus.cc:2258] T 57ddc99075d1455fbf3bf52d35641a04 P 74b98e19928d4abdbd1e062eddc64d68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:57.798749 18885 raft_consensus.cc:2258] T 57ddc99075d1455fbf3bf52d35641a04 P 7201b2b1dba94a56929d3723f0acd0a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:57.801229 18884 ts_tablet_manager.cc:1905] T 57ddc99075d1455fbf3bf52d35641a04 P 74b98e19928d4abdbd1e062eddc64d68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:57.802834 18885 ts_tablet_manager.cc:1905] T 57ddc99075d1455fbf3bf52d35641a04 P 7201b2b1dba94a56929d3723f0acd0a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:57.803997 18625 tablet_service.cc:1467] Processing CreateTablet for tablet 42c35f060e8d47dd886b2ef04472fec2 (DEFAULT_TABLE table=test-table [id=d9acc4f3f06d424980f41ca602450eaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:57.804418 18486 tablet_service.cc:1467] Processing CreateTablet for tablet 42c35f060e8d47dd886b2ef04472fec2 (DEFAULT_TABLE table=test-table [id=d9acc4f3f06d424980f41ca602450eaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:57.804598 18763 tablet_service.cc:1467] Processing CreateTablet for tablet 42c35f060e8d47dd886b2ef04472fec2 (DEFAULT_TABLE table=test-table [id=d9acc4f3f06d424980f41ca602450eaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:57.804814 18486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42c35f060e8d47dd886b2ef04472fec2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:57.805146 18763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42c35f060e8d47dd886b2ef04472fec2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:57.808761 18625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42c35f060e8d47dd886b2ef04472fec2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:57.811631 18861 tablet_bootstrap.cc:492] T 42c35f060e8d47dd886b2ef04472fec2 P 58f75de98c4c4930b6ee00f9c2c1a33f: Bootstrap starting.
I20260620 18:32:57.813876 18861 tablet_bootstrap.cc:654] T 42c35f060e8d47dd886b2ef04472fec2 P 58f75de98c4c4930b6ee00f9c2c1a33f: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:57.814533 18883 tablet_replica.cc:330] T 57ddc99075d1455fbf3bf52d35641a04 P 58f75de98c4c4930b6ee00f9c2c1a33f: stopping tablet replica
I20260620 18:32:57.814946 18883 raft_consensus.cc:2229] T 57ddc99075d1455fbf3bf52d35641a04 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:32:57.815732 18859 tablet_bootstrap.cc:492] T 42c35f060e8d47dd886b2ef04472fec2 P 74b98e19928d4abdbd1e062eddc64d68: Bootstrap starting.
I20260620 18:32:57.817639 18859 tablet_bootstrap.cc:654] T 42c35f060e8d47dd886b2ef04472fec2 P 74b98e19928d4abdbd1e062eddc64d68: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:57.817832 18883 raft_consensus.cc:2258] T 57ddc99075d1455fbf3bf52d35641a04 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:57.825414 18884 ts_tablet_manager.cc:1918] T 57ddc99075d1455fbf3bf52d35641a04 P 74b98e19928d4abdbd1e062eddc64d68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:32:57.825603 18884 log.cc:1196] T 57ddc99075d1455fbf3bf52d35641a04 P 74b98e19928d4abdbd1e062eddc64d68: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-1/wal/wals/57ddc99075d1455fbf3bf52d35641a04
I20260620 18:32:57.827585 18883 ts_tablet_manager.cc:1905] T 57ddc99075d1455fbf3bf52d35641a04 P 58f75de98c4c4930b6ee00f9c2c1a33f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:57.829404 18885 ts_tablet_manager.cc:1918] T 57ddc99075d1455fbf3bf52d35641a04 P 7201b2b1dba94a56929d3723f0acd0a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:32:57.829706 18885 log.cc:1196] T 57ddc99075d1455fbf3bf52d35641a04 P 7201b2b1dba94a56929d3723f0acd0a5: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-0/wal/wals/57ddc99075d1455fbf3bf52d35641a04
I20260620 18:32:57.830435 18885 ts_tablet_manager.cc:1939] T 57ddc99075d1455fbf3bf52d35641a04 P 7201b2b1dba94a56929d3723f0acd0a5: Deleting consensus metadata
I20260620 18:32:57.831065 18884 ts_tablet_manager.cc:1939] T 57ddc99075d1455fbf3bf52d35641a04 P 74b98e19928d4abdbd1e062eddc64d68: Deleting consensus metadata
I20260620 18:32:57.830649 18860 tablet_bootstrap.cc:492] T 42c35f060e8d47dd886b2ef04472fec2 P 7201b2b1dba94a56929d3723f0acd0a5: Bootstrap starting.
I20260620 18:32:57.832924 18860 tablet_bootstrap.cc:654] T 42c35f060e8d47dd886b2ef04472fec2 P 7201b2b1dba94a56929d3723f0acd0a5: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:57.834261 18334 catalog_manager.cc:4841] TS 7201b2b1dba94a56929d3723f0acd0a5 (127.10.131.1:35645): tablet 57ddc99075d1455fbf3bf52d35641a04 (table test-table [id=5755b67a74d24a8b9b8b10ea1a38274f]) successfully deleted
I20260620 18:32:57.834604 18336 catalog_manager.cc:4841] TS 74b98e19928d4abdbd1e062eddc64d68 (127.10.131.2:41595): tablet 57ddc99075d1455fbf3bf52d35641a04 (table test-table [id=5755b67a74d24a8b9b8b10ea1a38274f]) successfully deleted
I20260620 18:32:57.837843 18883 ts_tablet_manager.cc:1918] T 57ddc99075d1455fbf3bf52d35641a04 P 58f75de98c4c4930b6ee00f9c2c1a33f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:32:57.838070 18883 log.cc:1196] T 57ddc99075d1455fbf3bf52d35641a04 P 58f75de98c4c4930b6ee00f9c2c1a33f: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-2/wal/wals/57ddc99075d1455fbf3bf52d35641a04
I20260620 18:32:57.839723 18883 ts_tablet_manager.cc:1939] T 57ddc99075d1455fbf3bf52d35641a04 P 58f75de98c4c4930b6ee00f9c2c1a33f: Deleting consensus metadata
I20260620 18:32:57.841367 18334 catalog_manager.cc:4841] TS 58f75de98c4c4930b6ee00f9c2c1a33f (127.10.131.3:42999): tablet 57ddc99075d1455fbf3bf52d35641a04 (table test-table [id=5755b67a74d24a8b9b8b10ea1a38274f]) successfully deleted
I20260620 18:32:57.843050 18861 tablet_bootstrap.cc:492] T 42c35f060e8d47dd886b2ef04472fec2 P 58f75de98c4c4930b6ee00f9c2c1a33f: No bootstrap required, opened a new log
I20260620 18:32:57.843071 18860 tablet_bootstrap.cc:492] T 42c35f060e8d47dd886b2ef04472fec2 P 7201b2b1dba94a56929d3723f0acd0a5: No bootstrap required, opened a new log
I20260620 18:32:57.843232 18860 ts_tablet_manager.cc:1397] T 42c35f060e8d47dd886b2ef04472fec2 P 7201b2b1dba94a56929d3723f0acd0a5: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.009s
I20260620 18:32:57.843256 18861 ts_tablet_manager.cc:1397] T 42c35f060e8d47dd886b2ef04472fec2 P 58f75de98c4c4930b6ee00f9c2c1a33f: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.009s
I20260620 18:32:57.843850 18859 tablet_bootstrap.cc:492] T 42c35f060e8d47dd886b2ef04472fec2 P 74b98e19928d4abdbd1e062eddc64d68: No bootstrap required, opened a new log
I20260620 18:32:57.844036 18859 ts_tablet_manager.cc:1397] T 42c35f060e8d47dd886b2ef04472fec2 P 74b98e19928d4abdbd1e062eddc64d68: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.006s
I20260620 18:32:57.844130 18861 raft_consensus.cc:348] T 42c35f060e8d47dd886b2ef04472fec2 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } }
I20260620 18:32:57.844415 18861 raft_consensus.cc:374] T 42c35f060e8d47dd886b2ef04472fec2 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:57.844242 18860 raft_consensus.cc:348] T 42c35f060e8d47dd886b2ef04472fec2 P 7201b2b1dba94a56929d3723f0acd0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } }
I20260620 18:32:57.844558 18861 raft_consensus.cc:729] T 42c35f060e8d47dd886b2ef04472fec2 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f75de98c4c4930b6ee00f9c2c1a33f, State: Initialized, Role: FOLLOWER
I20260620 18:32:57.844626 18860 raft_consensus.cc:374] T 42c35f060e8d47dd886b2ef04472fec2 P 7201b2b1dba94a56929d3723f0acd0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:57.844816 18860 raft_consensus.cc:729] T 42c35f060e8d47dd886b2ef04472fec2 P 7201b2b1dba94a56929d3723f0acd0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7201b2b1dba94a56929d3723f0acd0a5, State: Initialized, Role: FOLLOWER
I20260620 18:32:57.844878 18859 raft_consensus.cc:348] T 42c35f060e8d47dd886b2ef04472fec2 P 74b98e19928d4abdbd1e062eddc64d68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } }
I20260620 18:32:57.844951 18861 consensus_queue.cc:260] T 42c35f060e8d47dd886b2ef04472fec2 P 58f75de98c4c4930b6ee00f9c2c1a33f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } }
I20260620 18:32:57.845136 18859 raft_consensus.cc:374] T 42c35f060e8d47dd886b2ef04472fec2 P 74b98e19928d4abdbd1e062eddc64d68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:57.845256 18859 raft_consensus.cc:729] T 42c35f060e8d47dd886b2ef04472fec2 P 74b98e19928d4abdbd1e062eddc64d68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74b98e19928d4abdbd1e062eddc64d68, State: Initialized, Role: FOLLOWER
I20260620 18:32:57.845620 18861 ts_tablet_manager.cc:1428] T 42c35f060e8d47dd886b2ef04472fec2 P 58f75de98c4c4930b6ee00f9c2c1a33f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:32:57.845480 18860 consensus_queue.cc:260] T 42c35f060e8d47dd886b2ef04472fec2 P 7201b2b1dba94a56929d3723f0acd0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } }
I20260620 18:32:57.845605 18859 consensus_queue.cc:260] T 42c35f060e8d47dd886b2ef04472fec2 P 74b98e19928d4abdbd1e062eddc64d68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } }
I20260620 18:32:57.848483 18859 ts_tablet_manager.cc:1428] T 42c35f060e8d47dd886b2ef04472fec2 P 74b98e19928d4abdbd1e062eddc64d68: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260620 18:32:57.848955 18860 ts_tablet_manager.cc:1428] T 42c35f060e8d47dd886b2ef04472fec2 P 7201b2b1dba94a56929d3723f0acd0a5: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20260620 18:32:57.880316 18867 raft_consensus.cc:482] T 42c35f060e8d47dd886b2ef04472fec2 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:32:57.880542 18867 raft_consensus.cc:504] T 42c35f060e8d47dd886b2ef04472fec2 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } }
I20260620 18:32:57.882094 18867 leader_election.cc:288] T 42c35f060e8d47dd886b2ef04472fec2 P 58f75de98c4c4930b6ee00f9c2c1a33f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7201b2b1dba94a56929d3723f0acd0a5 (127.10.131.1:35645), 74b98e19928d4abdbd1e062eddc64d68 (127.10.131.2:41595)
I20260620 18:32:57.890678 18856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:57.882644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:48875 (local address 127.10.131.2:41595)
0620 18:32:57.882833 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:57.882839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:57.883872 (+ 1033us) server_negotiation.cc:407] Connection header received
0620 18:32:57.884056 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:57.884064 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:57.884153 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:57.884242 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:57.884999 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:57.885717 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:57.888329 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:57.889614 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:57.889781 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:57.889853 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:57.890227 (+ 374us) server_negotiation.cc:299] Negotiation successful
0620 18:32:57.890364 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 18:32:57.890919 18890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:57.882366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:35645 (local address 127.10.131.3:34289)
0620 18:32:57.883470 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:57.883495 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:57.883714 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:57.884102 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:57.884107 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:57.884129 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:57.884494 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:57.884505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:57.886029 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:57.886039 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:57.888502 (+ 2463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:57.888518 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:57.890155 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:57.890170 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:57.890329 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:57.890351 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:32:57.890453 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 18:32:57.890577 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":962,"thread_start_us":1129,"threads_started":1}
I20260620 18:32:57.891376 18889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:57.882517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:41595 (local address 127.10.131.3:48875)
0620 18:32:57.883794 (+ 1277us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:57.883810 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:57.883903 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:57.884404 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:57.884409 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:57.884437 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:57.884808 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:57.884816 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:57.885850 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:57.885858 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:57.888190 (+ 2332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:57.888201 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:57.889824 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:57.889831 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:57.890015 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:57.890051 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 18:32:57.890860 (+ 809us) client_negotiation.cc:240] Negotiation successful
0620 18:32:57.890997 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1149,"mutex_wait_us":5,"thread_start_us":658,"threads_started":1}
I20260620 18:32:57.891408 18858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:57.882825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:34289 (local address 127.10.131.1:35645)
0620 18:32:57.883008 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:57.883017 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:57.883590 (+ 573us) server_negotiation.cc:407] Connection header received
0620 18:32:57.883880 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:57.883886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:57.883978 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:57.884081 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:57.884666 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:57.885892 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:57.888680 (+ 2788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:57.889937 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:57.890104 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:57.890184 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:57.890479 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 18:32:57.890604 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 18:32:57.892189 18650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42c35f060e8d47dd886b2ef04472fec2" candidate_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74b98e19928d4abdbd1e062eddc64d68" is_pre_election: true
I20260620 18:32:57.892428 18650 raft_consensus.cc:2454] T 42c35f060e8d47dd886b2ef04472fec2 P 74b98e19928d4abdbd1e062eddc64d68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f75de98c4c4930b6ee00f9c2c1a33f in term 0.
I20260620 18:32:57.892429 18511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42c35f060e8d47dd886b2ef04472fec2" candidate_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7201b2b1dba94a56929d3723f0acd0a5" is_pre_election: true
I20260620 18:32:57.892904 18718 leader_election.cc:302] T 42c35f060e8d47dd886b2ef04472fec2 P 58f75de98c4c4930b6ee00f9c2c1a33f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f75de98c4c4930b6ee00f9c2c1a33f, 74b98e19928d4abdbd1e062eddc64d68; no voters:
I20260620 18:32:57.892933 18511 raft_consensus.cc:2454] T 42c35f060e8d47dd886b2ef04472fec2 P 7201b2b1dba94a56929d3723f0acd0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f75de98c4c4930b6ee00f9c2c1a33f in term 0.
I20260620 18:32:57.893229 18867 raft_consensus.cc:2781] T 42c35f060e8d47dd886b2ef04472fec2 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:32:57.893344 18867 raft_consensus.cc:482] T 42c35f060e8d47dd886b2ef04472fec2 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:32:57.893466 18867 raft_consensus.cc:3037] T 42c35f060e8d47dd886b2ef04472fec2 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:57.895521 18867 raft_consensus.cc:504] T 42c35f060e8d47dd886b2ef04472fec2 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } }
I20260620 18:32:57.896111 18867 leader_election.cc:288] T 42c35f060e8d47dd886b2ef04472fec2 P 58f75de98c4c4930b6ee00f9c2c1a33f [CANDIDATE]: Term 1 election: Requested vote from peers 7201b2b1dba94a56929d3723f0acd0a5 (127.10.131.1:35645), 74b98e19928d4abdbd1e062eddc64d68 (127.10.131.2:41595)
I20260620 18:32:57.896581 18650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42c35f060e8d47dd886b2ef04472fec2" candidate_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74b98e19928d4abdbd1e062eddc64d68"
I20260620 18:32:57.896595 18511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42c35f060e8d47dd886b2ef04472fec2" candidate_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7201b2b1dba94a56929d3723f0acd0a5"
I20260620 18:32:57.896772 18650 raft_consensus.cc:3037] T 42c35f060e8d47dd886b2ef04472fec2 P 74b98e19928d4abdbd1e062eddc64d68 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:57.896796 18511 raft_consensus.cc:3037] T 42c35f060e8d47dd886b2ef04472fec2 P 7201b2b1dba94a56929d3723f0acd0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:57.898905 18650 raft_consensus.cc:2454] T 42c35f060e8d47dd886b2ef04472fec2 P 74b98e19928d4abdbd1e062eddc64d68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f75de98c4c4930b6ee00f9c2c1a33f in term 1.
I20260620 18:32:57.899272 18718 leader_election.cc:302] T 42c35f060e8d47dd886b2ef04472fec2 P 58f75de98c4c4930b6ee00f9c2c1a33f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f75de98c4c4930b6ee00f9c2c1a33f, 74b98e19928d4abdbd1e062eddc64d68; no voters:
I20260620 18:32:57.899519 18867 raft_consensus.cc:2781] T 42c35f060e8d47dd886b2ef04472fec2 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:57.900079 18867 raft_consensus.cc:686] T 42c35f060e8d47dd886b2ef04472fec2 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 1 LEADER]: Becoming Leader. State: Replica: 58f75de98c4c4930b6ee00f9c2c1a33f, State: Running, Role: LEADER
I20260620 18:32:57.900141 18511 raft_consensus.cc:2454] T 42c35f060e8d47dd886b2ef04472fec2 P 7201b2b1dba94a56929d3723f0acd0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f75de98c4c4930b6ee00f9c2c1a33f in term 1.
I20260620 18:32:57.900614 18867 consensus_queue.cc:237] T 42c35f060e8d47dd886b2ef04472fec2 P 58f75de98c4c4930b6ee00f9c2c1a33f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } }
I20260620 18:32:57.903970 18347 catalog_manager.cc:5495] T 42c35f060e8d47dd886b2ef04472fec2 P 58f75de98c4c4930b6ee00f9c2c1a33f reported cstate change: term changed from 0 to 1, leader changed from <none> to 58f75de98c4c4930b6ee00f9c2c1a33f (127.10.131.3). New cstate: current_term: 1 leader_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } health_report { overall_health: HEALTHY } } }
Jun 20 18:32:57 dist-test-slave-cbn3 krb5kdc[18304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980374, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.62@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM
I20260620 18:32:57.965983 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:57.951208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50950 (local address 127.10.131.62:40381)
0620 18:32:57.951462 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:57.951476 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:57.951964 (+ 488us) server_negotiation.cc:407] Connection header received
0620 18:32:57.952134 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:57.952137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:57.952196 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:57.952308 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:57.953786 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:57.954579 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:57.956416 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:57.956805 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:57.956968 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:57.960788 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:57.960793 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:57.960796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:57.960824 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:57.963281 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:57.964072 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:57.964076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:57.964079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:57.964250 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:57.964633 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:57.964636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:57.964638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:57.964983 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:57.965105 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:57.965560 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 18:32:57.965731 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260620 18:32:57.966341 18898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:57.951219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:40381 (local address 127.0.0.1:50950)
0620 18:32:57.951856 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:57.951877 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:57.952021 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:57.952363 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:57.952368 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:57.953293 (+ 925us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:32:57.953647 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:57.953659 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:57.954704 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:57.954710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:57.956277 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:57.956291 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:57.957041 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:57.957049 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:57.957219 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:57.957885 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:32:57.957909 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:32:57.960593 (+ 2684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:32:57.963393 (+ 2800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:57.963404 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:57.963412 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:57.963968 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:57.964334 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:32:57.964339 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:32:57.964343 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:32:57.964518 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:32:57.965112 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:32:57.965120 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:32:57.965296 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 18:32:57.965491 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 18:32:57.965905 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":117,"thread_start_us":361,"threads_started":1}
I20260620 18:32:57.971179 18347 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:50950:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 18:32:57.972433 18347 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:32:57.979090 18486 tablet_service.cc:1467] Processing CreateTablet for tablet 44c212f0e62b401d8f5ac24bad182829 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2773bfe73abe4cac840eb2d4bb5641a3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:32:57.979249 18625 tablet_service.cc:1467] Processing CreateTablet for tablet 44c212f0e62b401d8f5ac24bad182829 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2773bfe73abe4cac840eb2d4bb5641a3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:32:57.979661 18486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44c212f0e62b401d8f5ac24bad182829. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:57.979803 18625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44c212f0e62b401d8f5ac24bad182829. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:57.981354 18764 tablet_service.cc:1467] Processing CreateTablet for tablet 44c212f0e62b401d8f5ac24bad182829 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2773bfe73abe4cac840eb2d4bb5641a3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:32:57.981870 18764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44c212f0e62b401d8f5ac24bad182829. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:57.986852 18859 tablet_bootstrap.cc:492] T 44c212f0e62b401d8f5ac24bad182829 P 74b98e19928d4abdbd1e062eddc64d68: Bootstrap starting.
I20260620 18:32:57.988534 18859 tablet_bootstrap.cc:654] T 44c212f0e62b401d8f5ac24bad182829 P 74b98e19928d4abdbd1e062eddc64d68: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:57.988778 18861 tablet_bootstrap.cc:492] T 44c212f0e62b401d8f5ac24bad182829 P 58f75de98c4c4930b6ee00f9c2c1a33f: Bootstrap starting.
I20260620 18:32:57.990252 18860 tablet_bootstrap.cc:492] T 44c212f0e62b401d8f5ac24bad182829 P 7201b2b1dba94a56929d3723f0acd0a5: Bootstrap starting.
I20260620 18:32:57.990361 18861 tablet_bootstrap.cc:654] T 44c212f0e62b401d8f5ac24bad182829 P 58f75de98c4c4930b6ee00f9c2c1a33f: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:57.991739 18860 tablet_bootstrap.cc:654] T 44c212f0e62b401d8f5ac24bad182829 P 7201b2b1dba94a56929d3723f0acd0a5: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:57.992580 18861 tablet_bootstrap.cc:492] T 44c212f0e62b401d8f5ac24bad182829 P 58f75de98c4c4930b6ee00f9c2c1a33f: No bootstrap required, opened a new log
I20260620 18:32:57.992715 18861 ts_tablet_manager.cc:1397] T 44c212f0e62b401d8f5ac24bad182829 P 58f75de98c4c4930b6ee00f9c2c1a33f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 18:32:57.993280 18859 tablet_bootstrap.cc:492] T 44c212f0e62b401d8f5ac24bad182829 P 74b98e19928d4abdbd1e062eddc64d68: No bootstrap required, opened a new log
I20260620 18:32:57.993454 18859 ts_tablet_manager.cc:1397] T 44c212f0e62b401d8f5ac24bad182829 P 74b98e19928d4abdbd1e062eddc64d68: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20260620 18:32:57.993418 18861 raft_consensus.cc:348] T 44c212f0e62b401d8f5ac24bad182829 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } }
I20260620 18:32:57.993713 18861 raft_consensus.cc:374] T 44c212f0e62b401d8f5ac24bad182829 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:57.993796 18861 raft_consensus.cc:729] T 44c212f0e62b401d8f5ac24bad182829 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f75de98c4c4930b6ee00f9c2c1a33f, State: Initialized, Role: FOLLOWER
I20260620 18:32:57.994026 18861 consensus_queue.cc:260] T 44c212f0e62b401d8f5ac24bad182829 P 58f75de98c4c4930b6ee00f9c2c1a33f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } }
I20260620 18:32:57.994140 18859 raft_consensus.cc:348] T 44c212f0e62b401d8f5ac24bad182829 P 74b98e19928d4abdbd1e062eddc64d68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } }
I20260620 18:32:57.994422 18867 tablet_replica.cc:439] T 44c212f0e62b401d8f5ac24bad182829 P 58f75de98c4c4930b6ee00f9c2c1a33f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } } }
I20260620 18:32:57.994699 18867 tablet_replica.cc:442] T 44c212f0e62b401d8f5ac24bad182829 P 58f75de98c4c4930b6ee00f9c2c1a33f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:32:57.995034 18859 raft_consensus.cc:374] T 44c212f0e62b401d8f5ac24bad182829 P 74b98e19928d4abdbd1e062eddc64d68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:57.995184 18859 raft_consensus.cc:729] T 44c212f0e62b401d8f5ac24bad182829 P 74b98e19928d4abdbd1e062eddc64d68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74b98e19928d4abdbd1e062eddc64d68, State: Initialized, Role: FOLLOWER
I20260620 18:32:57.995604 18861 ts_tablet_manager.cc:1428] T 44c212f0e62b401d8f5ac24bad182829 P 58f75de98c4c4930b6ee00f9c2c1a33f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260620 18:32:57.995510 18859 consensus_queue.cc:260] T 44c212f0e62b401d8f5ac24bad182829 P 74b98e19928d4abdbd1e062eddc64d68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } }
I20260620 18:32:57.996091 18865 tablet_replica.cc:439] T 44c212f0e62b401d8f5ac24bad182829 P 74b98e19928d4abdbd1e062eddc64d68: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } } }
I20260620 18:32:57.996450 18865 tablet_replica.cc:442] T 44c212f0e62b401d8f5ac24bad182829 P 74b98e19928d4abdbd1e062eddc64d68: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:32:57.997120 18859 ts_tablet_manager.cc:1428] T 44c212f0e62b401d8f5ac24bad182829 P 74b98e19928d4abdbd1e062eddc64d68: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260620 18:32:57.998694 18860 tablet_bootstrap.cc:492] T 44c212f0e62b401d8f5ac24bad182829 P 7201b2b1dba94a56929d3723f0acd0a5: No bootstrap required, opened a new log
I20260620 18:32:57.998873 18860 ts_tablet_manager.cc:1397] T 44c212f0e62b401d8f5ac24bad182829 P 7201b2b1dba94a56929d3723f0acd0a5: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260620 18:32:57.999620 18860 raft_consensus.cc:348] T 44c212f0e62b401d8f5ac24bad182829 P 7201b2b1dba94a56929d3723f0acd0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } }
I20260620 18:32:57.999903 18860 raft_consensus.cc:374] T 44c212f0e62b401d8f5ac24bad182829 P 7201b2b1dba94a56929d3723f0acd0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:57.999986 18860 raft_consensus.cc:729] T 44c212f0e62b401d8f5ac24bad182829 P 7201b2b1dba94a56929d3723f0acd0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7201b2b1dba94a56929d3723f0acd0a5, State: Initialized, Role: FOLLOWER
I20260620 18:32:58.000336 18860 consensus_queue.cc:260] T 44c212f0e62b401d8f5ac24bad182829 P 7201b2b1dba94a56929d3723f0acd0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } }
I20260620 18:32:58.000998 18860 ts_tablet_manager.cc:1428] T 44c212f0e62b401d8f5ac24bad182829 P 7201b2b1dba94a56929d3723f0acd0a5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:32:58.000748 18879 tablet_replica.cc:439] T 44c212f0e62b401d8f5ac24bad182829 P 7201b2b1dba94a56929d3723f0acd0a5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } } }
I20260620 18:32:58.001166 18879 tablet_replica.cc:442] T 44c212f0e62b401d8f5ac24bad182829 P 7201b2b1dba94a56929d3723f0acd0a5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:32:58.105129 18865 raft_consensus.cc:482] T 44c212f0e62b401d8f5ac24bad182829 P 74b98e19928d4abdbd1e062eddc64d68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:32:58.105329 18865 raft_consensus.cc:504] T 44c212f0e62b401d8f5ac24bad182829 P 74b98e19928d4abdbd1e062eddc64d68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } }
I20260620 18:32:58.106945 18865 leader_election.cc:288] T 44c212f0e62b401d8f5ac24bad182829 P 74b98e19928d4abdbd1e062eddc64d68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7201b2b1dba94a56929d3723f0acd0a5 (127.10.131.1:35645), 58f75de98c4c4930b6ee00f9c2c1a33f (127.10.131.3:42999)
I20260620 18:32:58.116083 18902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:58.107256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:35645 (local address 127.10.131.2:49645)
0620 18:32:58.108167 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:58.108198 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:58.108388 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:58.109063 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:58.109070 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:58.109105 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:58.109604 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:58.109618 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:58.111122 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:58.111128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:58.113494 (+ 2366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:58.113507 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:58.115266 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:58.115272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:58.115440 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:58.115483 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 18:32:58.115573 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 18:32:58.115706 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"mutex_wait_us":22,"thread_start_us":641,"threads_started":1}
I20260620 18:32:58.116122 18858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:58.107227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:49645 (local address 127.10.131.1:35645)
0620 18:32:58.107413 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:58.107419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:58.108274 (+ 855us) server_negotiation.cc:407] Connection header received
0620 18:32:58.108613 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:58.108622 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:58.108716 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:58.108790 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:58.109824 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:58.110966 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:58.113659 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:58.114997 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:58.115203 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:58.115279 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:58.115634 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 18:32:58.115783 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:32:58.116217 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:58.108435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:58765 (local address 127.10.131.3:42999)
0620 18:32:58.108618 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:58.108623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:58.110313 (+ 1690us) server_negotiation.cc:407] Connection header received
0620 18:32:58.110478 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:58.110481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:58.110533 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:58.110622 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:32:58.111088 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:58.112233 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:58.114283 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:58.115060 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:58.115278 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:58.115349 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:58.115804 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 18:32:58.115937 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 18:32:58.116992 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:58.108303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:42999 (local address 127.10.131.2:58765)
0620 18:32:58.110216 (+ 1913us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:58.110234 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:58.110355 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:58.110631 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:58.110635 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:58.110656 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:32:58.110955 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:58.110962 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:58.112379 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:58.112384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:58.114143 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:58.114153 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:58.115482 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:58.115488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:58.115641 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:58.115670 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 18:32:58.116563 (+ 893us) client_negotiation.cc:240] Negotiation successful
0620 18:32:58.116665 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1815,"mutex_wait_us":1,"thread_start_us":1635,"threads_started":1}
I20260620 18:32:58.117039 18511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c212f0e62b401d8f5ac24bad182829" candidate_uuid: "74b98e19928d4abdbd1e062eddc64d68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7201b2b1dba94a56929d3723f0acd0a5" is_pre_election: true
I20260620 18:32:58.117354 18511 raft_consensus.cc:2454] T 44c212f0e62b401d8f5ac24bad182829 P 7201b2b1dba94a56929d3723f0acd0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74b98e19928d4abdbd1e062eddc64d68 in term 0.
I20260620 18:32:58.117632 18785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c212f0e62b401d8f5ac24bad182829" candidate_uuid: "74b98e19928d4abdbd1e062eddc64d68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" is_pre_election: true
I20260620 18:32:58.117872 18785 raft_consensus.cc:2454] T 44c212f0e62b401d8f5ac24bad182829 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74b98e19928d4abdbd1e062eddc64d68 in term 0.
I20260620 18:32:58.117837 18577 leader_election.cc:302] T 44c212f0e62b401d8f5ac24bad182829 P 74b98e19928d4abdbd1e062eddc64d68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7201b2b1dba94a56929d3723f0acd0a5, 74b98e19928d4abdbd1e062eddc64d68; no voters:
I20260620 18:32:58.118283 18865 raft_consensus.cc:2781] T 44c212f0e62b401d8f5ac24bad182829 P 74b98e19928d4abdbd1e062eddc64d68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:32:58.118428 18865 raft_consensus.cc:482] T 44c212f0e62b401d8f5ac24bad182829 P 74b98e19928d4abdbd1e062eddc64d68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:32:58.118543 18865 raft_consensus.cc:3037] T 44c212f0e62b401d8f5ac24bad182829 P 74b98e19928d4abdbd1e062eddc64d68 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:58.121701 18865 raft_consensus.cc:504] T 44c212f0e62b401d8f5ac24bad182829 P 74b98e19928d4abdbd1e062eddc64d68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } }
I20260620 18:32:58.122411 18865 leader_election.cc:288] T 44c212f0e62b401d8f5ac24bad182829 P 74b98e19928d4abdbd1e062eddc64d68 [CANDIDATE]: Term 1 election: Requested vote from peers 7201b2b1dba94a56929d3723f0acd0a5 (127.10.131.1:35645), 58f75de98c4c4930b6ee00f9c2c1a33f (127.10.131.3:42999)
I20260620 18:32:58.122922 18511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c212f0e62b401d8f5ac24bad182829" candidate_uuid: "74b98e19928d4abdbd1e062eddc64d68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7201b2b1dba94a56929d3723f0acd0a5"
I20260620 18:32:58.122953 18785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c212f0e62b401d8f5ac24bad182829" candidate_uuid: "74b98e19928d4abdbd1e062eddc64d68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f"
I20260620 18:32:58.123147 18785 raft_consensus.cc:3037] T 44c212f0e62b401d8f5ac24bad182829 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:58.123171 18511 raft_consensus.cc:3037] T 44c212f0e62b401d8f5ac24bad182829 P 7201b2b1dba94a56929d3723f0acd0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:58.125180 18785 raft_consensus.cc:2454] T 44c212f0e62b401d8f5ac24bad182829 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74b98e19928d4abdbd1e062eddc64d68 in term 1.
I20260620 18:32:58.125576 18577 leader_election.cc:302] T 44c212f0e62b401d8f5ac24bad182829 P 74b98e19928d4abdbd1e062eddc64d68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f75de98c4c4930b6ee00f9c2c1a33f, 74b98e19928d4abdbd1e062eddc64d68; no voters:
I20260620 18:32:58.125818 18865 raft_consensus.cc:2781] T 44c212f0e62b401d8f5ac24bad182829 P 74b98e19928d4abdbd1e062eddc64d68 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:58.126272 18511 raft_consensus.cc:2454] T 44c212f0e62b401d8f5ac24bad182829 P 7201b2b1dba94a56929d3723f0acd0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74b98e19928d4abdbd1e062eddc64d68 in term 1.
I20260620 18:32:58.126318 18865 raft_consensus.cc:686] T 44c212f0e62b401d8f5ac24bad182829 P 74b98e19928d4abdbd1e062eddc64d68 [term 1 LEADER]: Becoming Leader. State: Replica: 74b98e19928d4abdbd1e062eddc64d68, State: Running, Role: LEADER
I20260620 18:32:58.126747 18865 consensus_queue.cc:237] T 44c212f0e62b401d8f5ac24bad182829 P 74b98e19928d4abdbd1e062eddc64d68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } }
I20260620 18:32:58.129024 18904 tablet_replica.cc:439] T 44c212f0e62b401d8f5ac24bad182829 P 74b98e19928d4abdbd1e062eddc64d68: TxnStatusTablet state changed. Reason: New leader 74b98e19928d4abdbd1e062eddc64d68. Latest consensus state: current_term: 1 leader_uuid: "74b98e19928d4abdbd1e062eddc64d68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } } }
I20260620 18:32:58.129403 18904 tablet_replica.cc:442] T 44c212f0e62b401d8f5ac24bad182829 P 74b98e19928d4abdbd1e062eddc64d68: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:32:58.130142 18906 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 18:32:58.130364 18906 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 18:32:58.130448 18349 catalog_manager.cc:5495] T 44c212f0e62b401d8f5ac24bad182829 P 74b98e19928d4abdbd1e062eddc64d68 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74b98e19928d4abdbd1e062eddc64d68 (127.10.131.2). New cstate: current_term: 1 leader_uuid: "74b98e19928d4abdbd1e062eddc64d68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } health_report { overall_health: UNKNOWN } } }
I20260620 18:32:58.158361 18856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:58.151264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42926 (local address 127.10.131.2:41595)
0620 18:32:58.151435 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:58.151440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:58.151595 (+ 155us) server_negotiation.cc:407] Connection header received
0620 18:32:58.151877 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:58.151883 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:58.151985 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:58.152123 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:32:58.152539 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:58.153744 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:58.155634 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:58.156300 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:58.156442 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:58.157024 (+ 582us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:32:58.157700 (+ 676us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:32:58.157830 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:58.157973 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 18:32:58.158063 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:32:58.158390 18898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:58.151180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:41595 (local address 127.0.0.1:42926)
0620 18:32:58.151492 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:58.151512 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:58.151670 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:58.152180 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:58.152185 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:58.152198 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:32:58.152399 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:58.152410 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:58.153902 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:58.153908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:58.155476 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:58.155491 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:58.156487 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:58.156493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:58.156737 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:58.156808 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:32:58.157824 (+ 1016us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:32:58.157846 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:32:58.157942 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 18:32:58.158096 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260620 18:32:58.163054 18785 raft_consensus.cc:1261] T 44c212f0e62b401d8f5ac24bad182829 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 1 FOLLOWER]: Refusing update from remote peer 74b98e19928d4abdbd1e062eddc64d68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:58.163287 18511 raft_consensus.cc:1261] T 44c212f0e62b401d8f5ac24bad182829 P 7201b2b1dba94a56929d3723f0acd0a5 [term 1 FOLLOWER]: Refusing update from remote peer 74b98e19928d4abdbd1e062eddc64d68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:58.163868 18865 consensus_queue.cc:1035] T 44c212f0e62b401d8f5ac24bad182829 P 74b98e19928d4abdbd1e062eddc64d68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:58.164170 18904 consensus_queue.cc:1035] T 44c212f0e62b401d8f5ac24bad182829 P 74b98e19928d4abdbd1e062eddc64d68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:58.176380 18867 tablet_replica.cc:439] T 44c212f0e62b401d8f5ac24bad182829 P 58f75de98c4c4930b6ee00f9c2c1a33f: TxnStatusTablet state changed. Reason: New leader 74b98e19928d4abdbd1e062eddc64d68. Latest consensus state: current_term: 1 leader_uuid: "74b98e19928d4abdbd1e062eddc64d68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } } }
I20260620 18:32:58.176702 18867 tablet_replica.cc:442] T 44c212f0e62b401d8f5ac24bad182829 P 58f75de98c4c4930b6ee00f9c2c1a33f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:32:58.177711 18879 tablet_replica.cc:439] T 44c212f0e62b401d8f5ac24bad182829 P 7201b2b1dba94a56929d3723f0acd0a5: TxnStatusTablet state changed. Reason: New leader 74b98e19928d4abdbd1e062eddc64d68. Latest consensus state: current_term: 1 leader_uuid: "74b98e19928d4abdbd1e062eddc64d68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } } }
I20260620 18:32:58.178015 18879 tablet_replica.cc:442] T 44c212f0e62b401d8f5ac24bad182829 P 7201b2b1dba94a56929d3723f0acd0a5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:32:58.180339 18865 tablet_replica.cc:439] T 44c212f0e62b401d8f5ac24bad182829 P 74b98e19928d4abdbd1e062eddc64d68: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "74b98e19928d4abdbd1e062eddc64d68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } } }
I20260620 18:32:58.180644 18865 tablet_replica.cc:442] T 44c212f0e62b401d8f5ac24bad182829 P 74b98e19928d4abdbd1e062eddc64d68: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:32:58.183915 18874 mvcc.cc:205] Tried to move back new op lower bound from 7298991628947996672 to 7298991628810592256. Current Snapshot: MvccSnapshot[applied={T|T < 7298991628947996672 or (T in {7298991628947996672})}]
I20260620 18:32:58.186890 18879 tablet_replica.cc:439] T 44c212f0e62b401d8f5ac24bad182829 P 7201b2b1dba94a56929d3723f0acd0a5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "74b98e19928d4abdbd1e062eddc64d68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } } }
I20260620 18:32:58.187139 18879 tablet_replica.cc:442] T 44c212f0e62b401d8f5ac24bad182829 P 7201b2b1dba94a56929d3723f0acd0a5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:32:58.190448 18867 tablet_replica.cc:439] T 44c212f0e62b401d8f5ac24bad182829 P 58f75de98c4c4930b6ee00f9c2c1a33f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "74b98e19928d4abdbd1e062eddc64d68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } } }
I20260620 18:32:58.190742 18867 tablet_replica.cc:442] T 44c212f0e62b401d8f5ac24bad182829 P 58f75de98c4c4930b6ee00f9c2c1a33f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:32:58.192162 18904 tablet_replica.cc:439] T 44c212f0e62b401d8f5ac24bad182829 P 74b98e19928d4abdbd1e062eddc64d68: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "74b98e19928d4abdbd1e062eddc64d68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } } }
I20260620 18:32:58.192468 18904 tablet_replica.cc:442] T 44c212f0e62b401d8f5ac24bad182829 P 74b98e19928d4abdbd1e062eddc64d68: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:32:58.199895 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:58.187784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56832 (local address 127.10.131.3:42999)
0620 18:32:58.188467 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:58.188473 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:58.188490 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:32:58.188614 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:58.188622 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:58.188732 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:58.188910 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:32:58.189378 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:58.190616 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:58.196300 (+ 5684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:58.197048 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:58.197179 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:58.197839 (+ 660us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:32:58.198566 (+ 727us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:32:58.198793 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:58.199445 (+ 652us) server_negotiation.cc:299] Negotiation successful
0620 18:32:58.199534 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567}
I20260620 18:32:58.218228 18856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:58.211293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:38497 (local address 127.10.131.2:41595)
0620 18:32:58.211468 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:58.211474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:58.211696 (+ 222us) server_negotiation.cc:407] Connection header received
0620 18:32:58.211978 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:58.211985 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:58.212074 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:58.212177 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:32:58.212727 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:58.213876 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:58.215959 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:58.216693 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:58.216815 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:58.217365 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:32:58.217519 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:32:58.217621 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:58.217761 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 18:32:58.217857 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:32:58.218276 18914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:58.210952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:41595 (local address 127.10.131.3:38497)
0620 18:32:58.211584 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:58.211604 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:58.211811 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:58.212209 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:58.212215 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:58.212227 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:32:58.212556 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:58.212568 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:58.214013 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:58.214019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:58.215804 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:58.215819 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:58.216909 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:58.216918 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:58.217103 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:58.217174 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:32:58.217624 (+ 450us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:32:58.217642 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 18:32:58.217726 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 18:32:58.217875 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":524,"threads_started":1}
I20260620 18:32:58.230089 18650 raft_consensus.cc:1261] T 42c35f060e8d47dd886b2ef04472fec2 P 74b98e19928d4abdbd1e062eddc64d68 [term 1 FOLLOWER]: Refusing update from remote peer 58f75de98c4c4930b6ee00f9c2c1a33f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:58.230345 18511 raft_consensus.cc:1261] T 42c35f060e8d47dd886b2ef04472fec2 P 7201b2b1dba94a56929d3723f0acd0a5 [term 1 FOLLOWER]: Refusing update from remote peer 58f75de98c4c4930b6ee00f9c2c1a33f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:58.231077 18891 consensus_queue.cc:1035] T 42c35f060e8d47dd886b2ef04472fec2 P 58f75de98c4c4930b6ee00f9c2c1a33f [LEADER]: Connected to new peer: Peer: permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:58.231772 18867 consensus_queue.cc:1035] T 42c35f060e8d47dd886b2ef04472fec2 P 58f75de98c4c4930b6ee00f9c2c1a33f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:58.274724 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:58.267839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:42139 (local address 127.10.131.3:42999)
0620 18:32:58.268026 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:58.268034 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:58.268782 (+ 748us) server_negotiation.cc:407] Connection header received
0620 18:32:58.268980 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:58.268984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:58.269041 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:58.269120 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:32:58.269797 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:58.270659 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:58.272884 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:58.273321 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:58.273417 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:58.273913 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:32:58.274056 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:32:58.274164 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:58.274334 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 18:32:58.274405 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 18:32:58.274875 18918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:58.267860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:42999 (local address 127.10.131.2:42139)
0620 18:32:58.268672 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0620 18:32:58.268691 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:32:58.268841 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:32:58.269147 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:32:58.269151 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:32:58.269164 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:32:58.269620 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:58.269632 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:58.270781 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:58.270787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:58.272701 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:32:58.272716 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:32:58.273501 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:32:58.273508 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:32:58.273676 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:58.273743 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:32:58.274216 (+ 473us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:32:58.274229 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 18:32:58.274312 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 18:32:58.274445 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":425,"threads_started":1}
I20260620 18:32:58.308651 18347 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:32:58.308971 18347 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:32:58.312407 18347 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 012beec4185d4c8484308e45c3301e34: Sending DeleteTablet for 3 replicas of tablet 42c35f060e8d47dd886b2ef04472fec2
I20260620 18:32:58.313534 18486 tablet_service.cc:1514] Processing DeleteTablet for tablet 42c35f060e8d47dd886b2ef04472fec2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:58 UTC) from {username='oryx', principal='oryx/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:44582
I20260620 18:32:58.313810 18885 tablet_replica.cc:330] T 42c35f060e8d47dd886b2ef04472fec2 P 7201b2b1dba94a56929d3723f0acd0a5: stopping tablet replica
I20260620 18:32:58.314013 18885 raft_consensus.cc:2229] T 42c35f060e8d47dd886b2ef04472fec2 P 7201b2b1dba94a56929d3723f0acd0a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:32:58.314003 18764 tablet_service.cc:1514] Processing DeleteTablet for tablet 42c35f060e8d47dd886b2ef04472fec2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:58 UTC) from {username='oryx', principal='oryx/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:56828
I20260620 18:32:58.314209 18885 raft_consensus.cc:2258] T 42c35f060e8d47dd886b2ef04472fec2 P 7201b2b1dba94a56929d3723f0acd0a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:58.314250 18883 tablet_replica.cc:330] T 42c35f060e8d47dd886b2ef04472fec2 P 58f75de98c4c4930b6ee00f9c2c1a33f: stopping tablet replica
I20260620 18:32:58.314451 18883 raft_consensus.cc:2229] T 42c35f060e8d47dd886b2ef04472fec2 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:32:58.314714 18625 tablet_service.cc:1514] Processing DeleteTablet for tablet 42c35f060e8d47dd886b2ef04472fec2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:58 UTC) from {username='oryx', principal='oryx/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:42922
I20260620 18:32:58.314954 18884 tablet_replica.cc:330] T 42c35f060e8d47dd886b2ef04472fec2 P 74b98e19928d4abdbd1e062eddc64d68: stopping tablet replica
I20260620 18:32:58.314955 18883 raft_consensus.cc:2258] T 42c35f060e8d47dd886b2ef04472fec2 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:58.315183 18884 raft_consensus.cc:2229] T 42c35f060e8d47dd886b2ef04472fec2 P 74b98e19928d4abdbd1e062eddc64d68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:32:58.315382 18884 raft_consensus.cc:2258] T 42c35f060e8d47dd886b2ef04472fec2 P 74b98e19928d4abdbd1e062eddc64d68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:58.316299 18883 ts_tablet_manager.cc:1905] T 42c35f060e8d47dd886b2ef04472fec2 P 58f75de98c4c4930b6ee00f9c2c1a33f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:58.316645 18884 ts_tablet_manager.cc:1905] T 42c35f060e8d47dd886b2ef04472fec2 P 74b98e19928d4abdbd1e062eddc64d68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:58.323966 18884 ts_tablet_manager.cc:1918] T 42c35f060e8d47dd886b2ef04472fec2 P 74b98e19928d4abdbd1e062eddc64d68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:32:58.324167 18884 log.cc:1196] T 42c35f060e8d47dd886b2ef04472fec2 P 74b98e19928d4abdbd1e062eddc64d68: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-1/wal/wals/42c35f060e8d47dd886b2ef04472fec2
I20260620 18:32:58.324648 18884 ts_tablet_manager.cc:1939] T 42c35f060e8d47dd886b2ef04472fec2 P 74b98e19928d4abdbd1e062eddc64d68: Deleting consensus metadata
I20260620 18:32:58.325853 18336 catalog_manager.cc:4841] TS 74b98e19928d4abdbd1e062eddc64d68 (127.10.131.2:41595): tablet 42c35f060e8d47dd886b2ef04472fec2 (table test-table [id=d9acc4f3f06d424980f41ca602450eaf]) successfully deleted
I20260620 18:32:58.326066 18885 ts_tablet_manager.cc:1905] T 42c35f060e8d47dd886b2ef04472fec2 P 7201b2b1dba94a56929d3723f0acd0a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:58.326100 18883 ts_tablet_manager.cc:1918] T 42c35f060e8d47dd886b2ef04472fec2 P 58f75de98c4c4930b6ee00f9c2c1a33f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:32:58.326265 18883 log.cc:1196] T 42c35f060e8d47dd886b2ef04472fec2 P 58f75de98c4c4930b6ee00f9c2c1a33f: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-2/wal/wals/42c35f060e8d47dd886b2ef04472fec2
I20260620 18:32:58.326646 18883 ts_tablet_manager.cc:1939] T 42c35f060e8d47dd886b2ef04472fec2 P 58f75de98c4c4930b6ee00f9c2c1a33f: Deleting consensus metadata
I20260620 18:32:58.327531 18334 catalog_manager.cc:4841] TS 58f75de98c4c4930b6ee00f9c2c1a33f (127.10.131.3:42999): tablet 42c35f060e8d47dd886b2ef04472fec2 (table test-table [id=d9acc4f3f06d424980f41ca602450eaf]) successfully deleted
I20260620 18:32:58.331323 18885 ts_tablet_manager.cc:1918] T 42c35f060e8d47dd886b2ef04472fec2 P 7201b2b1dba94a56929d3723f0acd0a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:32:58.331496 18885 log.cc:1196] T 42c35f060e8d47dd886b2ef04472fec2 P 7201b2b1dba94a56929d3723f0acd0a5: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-0/wal/wals/42c35f060e8d47dd886b2ef04472fec2
I20260620 18:32:58.331892 18885 ts_tablet_manager.cc:1939] T 42c35f060e8d47dd886b2ef04472fec2 P 7201b2b1dba94a56929d3723f0acd0a5: Deleting consensus metadata
I20260620 18:32:58.332914 18334 catalog_manager.cc:4841] TS 7201b2b1dba94a56929d3723f0acd0a5 (127.10.131.1:35645): tablet 42c35f060e8d47dd886b2ef04472fec2 (table test-table [id=d9acc4f3f06d424980f41ca602450eaf]) successfully deleted
Jun 20 18:32:58 dist-test-slave-cbn3 krb5kdc[18304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 18:32:58 dist-test-slave-cbn3 krb5kdc[18304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:32:58 dist-test-slave-cbn3 krb5kdc[18304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 18:32:58.396962 18444 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:32:58.383471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50962 (local address 127.10.131.62:40381)
0620 18:32:58.383661 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:58.383668 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:58.384180 (+ 512us) server_negotiation.cc:407] Connection header received
0620 18:32:58.384346 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:58.384352 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:58.384440 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:58.384550 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:58.386208 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:58.387374 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:58.389313 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:58.389987 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:58.390111 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:58.396602 (+ 6491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50962: BlockingRecv error: recv got EOF from 127.0.0.1:50962 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 18:32:58 dist-test-slave-cbn3 krb5kdc[18304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM
I20260620 18:32:58.477712 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:58.446522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50968 (local address 127.10.131.62:40381)
0620 18:32:58.446722 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:58.446728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:58.458799 (+ 12071us) server_negotiation.cc:407] Connection header received
0620 18:32:58.459288 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:58.459294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:58.459401 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:58.459479 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:58.461388 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:58.462551 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:58.464694 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:58.465351 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:58.465444 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:58.470843 (+ 5399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:58.470850 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:58.470855 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:58.470903 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:58.474438 (+ 3535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:58.475284 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:58.475290 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:58.475294 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:58.475477 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:58.475921 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:58.475928 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:58.475931 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:58.476288 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:58.476530 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:58.477098 (+ 568us) server_negotiation.cc:299] Negotiation successful
0620 18:32:58.477334 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:32:58.482462 18347 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:32:58.483125 18347 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:32:58.491333 18486 tablet_service.cc:1467] Processing CreateTablet for tablet b0de64a506394e17a1919054b4f6e125 (DEFAULT_TABLE table=test-table [id=bea2f443d9574e9ea78e42f5391d64dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:58.491422 18625 tablet_service.cc:1467] Processing CreateTablet for tablet b0de64a506394e17a1919054b4f6e125 (DEFAULT_TABLE table=test-table [id=bea2f443d9574e9ea78e42f5391d64dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:58.491793 18486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0de64a506394e17a1919054b4f6e125. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:58.491835 18625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0de64a506394e17a1919054b4f6e125. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:58.497820 18764 tablet_service.cc:1467] Processing CreateTablet for tablet b0de64a506394e17a1919054b4f6e125 (DEFAULT_TABLE table=test-table [id=bea2f443d9574e9ea78e42f5391d64dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:58.498292 18764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0de64a506394e17a1919054b4f6e125. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:58.499567 18860 tablet_bootstrap.cc:492] T b0de64a506394e17a1919054b4f6e125 P 7201b2b1dba94a56929d3723f0acd0a5: Bootstrap starting.
I20260620 18:32:58.501199 18860 tablet_bootstrap.cc:654] T b0de64a506394e17a1919054b4f6e125 P 7201b2b1dba94a56929d3723f0acd0a5: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:58.509323 18860 tablet_bootstrap.cc:492] T b0de64a506394e17a1919054b4f6e125 P 7201b2b1dba94a56929d3723f0acd0a5: No bootstrap required, opened a new log
I20260620 18:32:58.509339 18936 tablet_bootstrap.cc:492] T b0de64a506394e17a1919054b4f6e125 P 58f75de98c4c4930b6ee00f9c2c1a33f: Bootstrap starting.
I20260620 18:32:58.509665 18860 ts_tablet_manager.cc:1397] T b0de64a506394e17a1919054b4f6e125 P 7201b2b1dba94a56929d3723f0acd0a5: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260620 18:32:58.510742 18860 raft_consensus.cc:348] T b0de64a506394e17a1919054b4f6e125 P 7201b2b1dba94a56929d3723f0acd0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } }
I20260620 18:32:58.511066 18936 tablet_bootstrap.cc:654] T b0de64a506394e17a1919054b4f6e125 P 58f75de98c4c4930b6ee00f9c2c1a33f: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:58.511220 18860 raft_consensus.cc:374] T b0de64a506394e17a1919054b4f6e125 P 7201b2b1dba94a56929d3723f0acd0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:58.511384 18860 raft_consensus.cc:729] T b0de64a506394e17a1919054b4f6e125 P 7201b2b1dba94a56929d3723f0acd0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7201b2b1dba94a56929d3723f0acd0a5, State: Initialized, Role: FOLLOWER
I20260620 18:32:58.511716 18860 consensus_queue.cc:260] T b0de64a506394e17a1919054b4f6e125 P 7201b2b1dba94a56929d3723f0acd0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } }
I20260620 18:32:58.513201 18860 ts_tablet_manager.cc:1428] T b0de64a506394e17a1919054b4f6e125 P 7201b2b1dba94a56929d3723f0acd0a5: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 18:32:58.513278 18935 tablet_bootstrap.cc:492] T b0de64a506394e17a1919054b4f6e125 P 74b98e19928d4abdbd1e062eddc64d68: Bootstrap starting.
I20260620 18:32:58.515065 18935 tablet_bootstrap.cc:654] T b0de64a506394e17a1919054b4f6e125 P 74b98e19928d4abdbd1e062eddc64d68: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:58.518442 18936 tablet_bootstrap.cc:492] T b0de64a506394e17a1919054b4f6e125 P 58f75de98c4c4930b6ee00f9c2c1a33f: No bootstrap required, opened a new log
I20260620 18:32:58.518589 18936 ts_tablet_manager.cc:1397] T b0de64a506394e17a1919054b4f6e125 P 58f75de98c4c4930b6ee00f9c2c1a33f: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20260620 18:32:58.519542 18935 tablet_bootstrap.cc:492] T b0de64a506394e17a1919054b4f6e125 P 74b98e19928d4abdbd1e062eddc64d68: No bootstrap required, opened a new log
I20260620 18:32:58.519431 18936 raft_consensus.cc:348] T b0de64a506394e17a1919054b4f6e125 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } }
I20260620 18:32:58.519734 18935 ts_tablet_manager.cc:1397] T b0de64a506394e17a1919054b4f6e125 P 74b98e19928d4abdbd1e062eddc64d68: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 18:32:58.519733 18936 raft_consensus.cc:374] T b0de64a506394e17a1919054b4f6e125 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:58.519833 18936 raft_consensus.cc:729] T b0de64a506394e17a1919054b4f6e125 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f75de98c4c4930b6ee00f9c2c1a33f, State: Initialized, Role: FOLLOWER
I20260620 18:32:58.520147 18936 consensus_queue.cc:260] T b0de64a506394e17a1919054b4f6e125 P 58f75de98c4c4930b6ee00f9c2c1a33f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } }
I20260620 18:32:58.520699 18935 raft_consensus.cc:348] T b0de64a506394e17a1919054b4f6e125 P 74b98e19928d4abdbd1e062eddc64d68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } }
I20260620 18:32:58.521104 18935 raft_consensus.cc:374] T b0de64a506394e17a1919054b4f6e125 P 74b98e19928d4abdbd1e062eddc64d68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:58.521265 18935 raft_consensus.cc:729] T b0de64a506394e17a1919054b4f6e125 P 74b98e19928d4abdbd1e062eddc64d68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74b98e19928d4abdbd1e062eddc64d68, State: Initialized, Role: FOLLOWER
I20260620 18:32:58.521575 18935 consensus_queue.cc:260] T b0de64a506394e17a1919054b4f6e125 P 74b98e19928d4abdbd1e062eddc64d68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } }
I20260620 18:32:58.522847 18936 ts_tablet_manager.cc:1428] T b0de64a506394e17a1919054b4f6e125 P 58f75de98c4c4930b6ee00f9c2c1a33f: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260620 18:32:58.523448 18935 ts_tablet_manager.cc:1428] T b0de64a506394e17a1919054b4f6e125 P 74b98e19928d4abdbd1e062eddc64d68: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 18:32:58.609241 18879 raft_consensus.cc:482] T b0de64a506394e17a1919054b4f6e125 P 7201b2b1dba94a56929d3723f0acd0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:32:58.609463 18879 raft_consensus.cc:504] T b0de64a506394e17a1919054b4f6e125 P 7201b2b1dba94a56929d3723f0acd0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } }
I20260620 18:32:58.610175 18879 leader_election.cc:288] T b0de64a506394e17a1919054b4f6e125 P 7201b2b1dba94a56929d3723f0acd0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74b98e19928d4abdbd1e062eddc64d68 (127.10.131.2:41595), 58f75de98c4c4930b6ee00f9c2c1a33f (127.10.131.3:42999)
I20260620 18:32:58.611888 18650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0de64a506394e17a1919054b4f6e125" candidate_uuid: "7201b2b1dba94a56929d3723f0acd0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74b98e19928d4abdbd1e062eddc64d68" is_pre_election: true
I20260620 18:32:58.612159 18650 raft_consensus.cc:2454] T b0de64a506394e17a1919054b4f6e125 P 74b98e19928d4abdbd1e062eddc64d68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201b2b1dba94a56929d3723f0acd0a5 in term 0.
I20260620 18:32:58.613013 18785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0de64a506394e17a1919054b4f6e125" candidate_uuid: "7201b2b1dba94a56929d3723f0acd0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" is_pre_election: true
I20260620 18:32:58.613456 18785 raft_consensus.cc:2454] T b0de64a506394e17a1919054b4f6e125 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201b2b1dba94a56929d3723f0acd0a5 in term 0.
I20260620 18:32:58.614043 18437 leader_election.cc:302] T b0de64a506394e17a1919054b4f6e125 P 7201b2b1dba94a56929d3723f0acd0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f75de98c4c4930b6ee00f9c2c1a33f, 7201b2b1dba94a56929d3723f0acd0a5; no voters:
I20260620 18:32:58.614954 18879 raft_consensus.cc:2781] T b0de64a506394e17a1919054b4f6e125 P 7201b2b1dba94a56929d3723f0acd0a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:32:58.615072 18879 raft_consensus.cc:482] T b0de64a506394e17a1919054b4f6e125 P 7201b2b1dba94a56929d3723f0acd0a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:32:58.615159 18879 raft_consensus.cc:3037] T b0de64a506394e17a1919054b4f6e125 P 7201b2b1dba94a56929d3723f0acd0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:58.619166 18879 raft_consensus.cc:504] T b0de64a506394e17a1919054b4f6e125 P 7201b2b1dba94a56929d3723f0acd0a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } }
I20260620 18:32:58.620489 18650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0de64a506394e17a1919054b4f6e125" candidate_uuid: "7201b2b1dba94a56929d3723f0acd0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74b98e19928d4abdbd1e062eddc64d68"
I20260620 18:32:58.620707 18650 raft_consensus.cc:3037] T b0de64a506394e17a1919054b4f6e125 P 74b98e19928d4abdbd1e062eddc64d68 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:58.624100 18650 raft_consensus.cc:2454] T b0de64a506394e17a1919054b4f6e125 P 74b98e19928d4abdbd1e062eddc64d68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201b2b1dba94a56929d3723f0acd0a5 in term 1.
I20260620 18:32:58.624675 18439 leader_election.cc:302] T b0de64a506394e17a1919054b4f6e125 P 7201b2b1dba94a56929d3723f0acd0a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7201b2b1dba94a56929d3723f0acd0a5, 74b98e19928d4abdbd1e062eddc64d68; no voters:
I20260620 18:32:58.625634 18879 leader_election.cc:288] T b0de64a506394e17a1919054b4f6e125 P 7201b2b1dba94a56929d3723f0acd0a5 [CANDIDATE]: Term 1 election: Requested vote from peers 74b98e19928d4abdbd1e062eddc64d68 (127.10.131.2:41595), 58f75de98c4c4930b6ee00f9c2c1a33f (127.10.131.3:42999)
I20260620 18:32:58.625826 18879 raft_consensus.cc:2781] T b0de64a506394e17a1919054b4f6e125 P 7201b2b1dba94a56929d3723f0acd0a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:58.625945 18879 raft_consensus.cc:686] T b0de64a506394e17a1919054b4f6e125 P 7201b2b1dba94a56929d3723f0acd0a5 [term 1 LEADER]: Becoming Leader. State: Replica: 7201b2b1dba94a56929d3723f0acd0a5, State: Running, Role: LEADER
I20260620 18:32:58.626272 18785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0de64a506394e17a1919054b4f6e125" candidate_uuid: "7201b2b1dba94a56929d3723f0acd0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f"
I20260620 18:32:58.626219 18879 consensus_queue.cc:237] T b0de64a506394e17a1919054b4f6e125 P 7201b2b1dba94a56929d3723f0acd0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } }
I20260620 18:32:58.626499 18785 raft_consensus.cc:3037] T b0de64a506394e17a1919054b4f6e125 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:58.629954 18785 raft_consensus.cc:2454] T b0de64a506394e17a1919054b4f6e125 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201b2b1dba94a56929d3723f0acd0a5 in term 1.
I20260620 18:32:58.641235 18347 catalog_manager.cc:5495] T b0de64a506394e17a1919054b4f6e125 P 7201b2b1dba94a56929d3723f0acd0a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7201b2b1dba94a56929d3723f0acd0a5 (127.10.131.1). New cstate: current_term: 1 leader_uuid: "7201b2b1dba94a56929d3723f0acd0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } health_report { overall_health: UNKNOWN } } }
I20260620 18:32:58.673663 18942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:58.667298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35084 (local address 127.10.131.1:35645)
0620 18:32:58.667795 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:58.667801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:58.667821 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:32:58.667896 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:58.667901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:58.667992 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:58.668104 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:32:58.668511 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:58.669654 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:58.671553 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:58.672208 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:58.672349 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:58.672826 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:32:58.672992 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:32:58.673088 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:58.673240 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 18:32:58.673327 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":233,"threads_started":1}
I20260620 18:32:58.676491 18785 raft_consensus.cc:1261] T b0de64a506394e17a1919054b4f6e125 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 1 FOLLOWER]: Refusing update from remote peer 7201b2b1dba94a56929d3723f0acd0a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:58.676698 18650 raft_consensus.cc:1261] T b0de64a506394e17a1919054b4f6e125 P 74b98e19928d4abdbd1e062eddc64d68 [term 1 FOLLOWER]: Refusing update from remote peer 7201b2b1dba94a56929d3723f0acd0a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:58.677314 18939 consensus_queue.cc:1035] T b0de64a506394e17a1919054b4f6e125 P 7201b2b1dba94a56929d3723f0acd0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:58.677716 18879 consensus_queue.cc:1035] T b0de64a506394e17a1919054b4f6e125 P 7201b2b1dba94a56929d3723f0acd0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:58.695616 18347 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:32:58.695977 18347 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:32:58.699436 18347 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 012beec4185d4c8484308e45c3301e34: Sending DeleteTablet for 3 replicas of tablet b0de64a506394e17a1919054b4f6e125
I20260620 18:32:58.700269 18486 tablet_service.cc:1514] Processing DeleteTablet for tablet b0de64a506394e17a1919054b4f6e125 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:58 UTC) from {username='oryx', principal='oryx/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:44582
I20260620 18:32:58.700635 18885 tablet_replica.cc:330] T b0de64a506394e17a1919054b4f6e125 P 7201b2b1dba94a56929d3723f0acd0a5: stopping tablet replica
I20260620 18:32:58.700752 18625 tablet_service.cc:1514] Processing DeleteTablet for tablet b0de64a506394e17a1919054b4f6e125 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:58 UTC) from {username='oryx', principal='oryx/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:42922
I20260620 18:32:58.700888 18885 raft_consensus.cc:2229] T b0de64a506394e17a1919054b4f6e125 P 7201b2b1dba94a56929d3723f0acd0a5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:32:58.701054 18884 tablet_replica.cc:330] T b0de64a506394e17a1919054b4f6e125 P 74b98e19928d4abdbd1e062eddc64d68: stopping tablet replica
I20260620 18:32:58.702024 18764 tablet_service.cc:1514] Processing DeleteTablet for tablet b0de64a506394e17a1919054b4f6e125 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:58 UTC) from {username='oryx', principal='oryx/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:56828
I20260620 18:32:58.702358 18883 tablet_replica.cc:330] T b0de64a506394e17a1919054b4f6e125 P 58f75de98c4c4930b6ee00f9c2c1a33f: stopping tablet replica
I20260620 18:32:58.702544 18883 raft_consensus.cc:2229] T b0de64a506394e17a1919054b4f6e125 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:32:58.702724 18883 pending_rounds.cc:62] T b0de64a506394e17a1919054b4f6e125 P 58f75de98c4c4930b6ee00f9c2c1a33f: Trying to abort 2 pending ops.
I20260620 18:32:58.702836 18883 pending_rounds.cc:68] T b0de64a506394e17a1919054b4f6e125 P 58f75de98c4c4930b6ee00f9c2c1a33f: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7298991631052865536 op_type: WRITE_OP write_request { tablet_id: "b0de64a506394e17a1919054b4f6e125" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\206\274\333\321\006\"8\n\ntest-admin\022*\n bea2f443d9574e9ea78e42f5391d64dd\020\001\030\001 \001(\001" signature: "<redacted>""Sh\242\252O\234\037\"\367\001\343N\025&WM\320g\2423r:\024\255\177\203\024\355\3600\2225\234\347\010G\306\'@Y\230\273\236\345\270!\n\031\321\267\265\010\260\300.3\375y\240\255\343f6\366" signing_key_seq_num: 0 } } request_id { client_id: "e0dcdae83c134b20b1ca20c97d34a3da" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260620 18:32:58.703687 18885 raft_consensus.cc:2258] T b0de64a506394e17a1919054b4f6e125 P 7201b2b1dba94a56929d3723f0acd0a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:58.703646 18883 pending_rounds.cc:68] T b0de64a506394e17a1919054b4f6e125 P 58f75de98c4c4930b6ee00f9c2c1a33f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7298991630855303168 op_type: NO_OP noop_request { }
I20260620 18:32:58.703790 18883 raft_consensus.cc:2866] T b0de64a506394e17a1919054b4f6e125 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260620 18:32:58.703943 18883 raft_consensus.cc:2258] T b0de64a506394e17a1919054b4f6e125 P 58f75de98c4c4930b6ee00f9c2c1a33f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:58.705130 18883 ts_tablet_manager.cc:1905] T b0de64a506394e17a1919054b4f6e125 P 58f75de98c4c4930b6ee00f9c2c1a33f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:58.705694 18349 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:32:58.706420 18349 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:32:58.707590 18884 raft_consensus.cc:2229] T b0de64a506394e17a1919054b4f6e125 P 74b98e19928d4abdbd1e062eddc64d68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:32:58.707676 18885 ts_tablet_manager.cc:1905] T b0de64a506394e17a1919054b4f6e125 P 7201b2b1dba94a56929d3723f0acd0a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:58.708102 18884 pending_rounds.cc:62] T b0de64a506394e17a1919054b4f6e125 P 74b98e19928d4abdbd1e062eddc64d68: Trying to abort 2 pending ops.
I20260620 18:32:58.708217 18884 pending_rounds.cc:68] T b0de64a506394e17a1919054b4f6e125 P 74b98e19928d4abdbd1e062eddc64d68: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7298991631052865536 op_type: WRITE_OP write_request { tablet_id: "b0de64a506394e17a1919054b4f6e125" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\206\274\333\321\006\"8\n\ntest-admin\022*\n bea2f443d9574e9ea78e42f5391d64dd\020\001\030\001 \001(\001" signature: "<redacted>""Sh\242\252O\234\037\"\367\001\343N\025&WM\320g\2423r:\024\255\177\203\024\355\3600\2225\234\347\010G\306\'@Y\230\273\236\345\270!\n\031\321\267\265\010\260\300.3\375y\240\255\343f6\366" signing_key_seq_num: 0 } } request_id { client_id: "e0dcdae83c134b20b1ca20c97d34a3da" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260620 18:32:58.709053 18884 pending_rounds.cc:68] T b0de64a506394e17a1919054b4f6e125 P 74b98e19928d4abdbd1e062eddc64d68: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7298991630855303168 op_type: NO_OP noop_request { }
I20260620 18:32:58.709231 18884 raft_consensus.cc:2866] T b0de64a506394e17a1919054b4f6e125 P 74b98e19928d4abdbd1e062eddc64d68 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260620 18:32:58.709390 18884 raft_consensus.cc:2258] T b0de64a506394e17a1919054b4f6e125 P 74b98e19928d4abdbd1e062eddc64d68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:58.714303 18885 ts_tablet_manager.cc:1918] T b0de64a506394e17a1919054b4f6e125 P 7201b2b1dba94a56929d3723f0acd0a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:32:58.714478 18885 log.cc:1196] T b0de64a506394e17a1919054b4f6e125 P 7201b2b1dba94a56929d3723f0acd0a5: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-0/wal/wals/b0de64a506394e17a1919054b4f6e125
I20260620 18:32:58.714548 18625 tablet_service.cc:1467] Processing CreateTablet for tablet 94e3a0a5a56846ac83f9a618a760e87a (DEFAULT_TABLE table=test-table [id=cb7165c353bf4b9d80b9077bbe9e6400]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:58.714891 18885 ts_tablet_manager.cc:1939] T b0de64a506394e17a1919054b4f6e125 P 7201b2b1dba94a56929d3723f0acd0a5: Deleting consensus metadata
I20260620 18:32:58.715039 18625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94e3a0a5a56846ac83f9a618a760e87a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:58.716540 18884 ts_tablet_manager.cc:1905] T b0de64a506394e17a1919054b4f6e125 P 74b98e19928d4abdbd1e062eddc64d68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:58.716997 18334 catalog_manager.cc:4841] TS 7201b2b1dba94a56929d3723f0acd0a5 (127.10.131.1:35645): tablet b0de64a506394e17a1919054b4f6e125 (table test-table [id=bea2f443d9574e9ea78e42f5391d64dd]) successfully deleted
I20260620 18:32:58.717432 18764 tablet_service.cc:1467] Processing CreateTablet for tablet 94e3a0a5a56846ac83f9a618a760e87a (DEFAULT_TABLE table=test-table [id=cb7165c353bf4b9d80b9077bbe9e6400]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:58.717613 18486 tablet_service.cc:1467] Processing CreateTablet for tablet 94e3a0a5a56846ac83f9a618a760e87a (DEFAULT_TABLE table=test-table [id=cb7165c353bf4b9d80b9077bbe9e6400]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:32:58.717839 18764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94e3a0a5a56846ac83f9a618a760e87a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:58.717967 18486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94e3a0a5a56846ac83f9a618a760e87a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:32:58.723950 18860 tablet_bootstrap.cc:492] T 94e3a0a5a56846ac83f9a618a760e87a P 7201b2b1dba94a56929d3723f0acd0a5: Bootstrap starting.
I20260620 18:32:58.725569 18860 tablet_bootstrap.cc:654] T 94e3a0a5a56846ac83f9a618a760e87a P 7201b2b1dba94a56929d3723f0acd0a5: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:58.727653 18935 tablet_bootstrap.cc:492] T 94e3a0a5a56846ac83f9a618a760e87a P 74b98e19928d4abdbd1e062eddc64d68: Bootstrap starting.
I20260620 18:32:58.728256 18860 tablet_bootstrap.cc:492] T 94e3a0a5a56846ac83f9a618a760e87a P 7201b2b1dba94a56929d3723f0acd0a5: No bootstrap required, opened a new log
I20260620 18:32:58.728482 18860 ts_tablet_manager.cc:1397] T 94e3a0a5a56846ac83f9a618a760e87a P 7201b2b1dba94a56929d3723f0acd0a5: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 18:32:58.729220 18935 tablet_bootstrap.cc:654] T 94e3a0a5a56846ac83f9a618a760e87a P 74b98e19928d4abdbd1e062eddc64d68: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:58.729255 18936 tablet_bootstrap.cc:492] T 94e3a0a5a56846ac83f9a618a760e87a P 58f75de98c4c4930b6ee00f9c2c1a33f: Bootstrap starting.
I20260620 18:32:58.730547 18860 raft_consensus.cc:348] T 94e3a0a5a56846ac83f9a618a760e87a P 7201b2b1dba94a56929d3723f0acd0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } }
I20260620 18:32:58.730824 18860 raft_consensus.cc:374] T 94e3a0a5a56846ac83f9a618a760e87a P 7201b2b1dba94a56929d3723f0acd0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:58.730912 18860 raft_consensus.cc:729] T 94e3a0a5a56846ac83f9a618a760e87a P 7201b2b1dba94a56929d3723f0acd0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7201b2b1dba94a56929d3723f0acd0a5, State: Initialized, Role: FOLLOWER
I20260620 18:32:58.730967 18936 tablet_bootstrap.cc:654] T 94e3a0a5a56846ac83f9a618a760e87a P 58f75de98c4c4930b6ee00f9c2c1a33f: Neither blocks nor log segments found. Creating new log.
I20260620 18:32:58.731276 18860 consensus_queue.cc:260] T 94e3a0a5a56846ac83f9a618a760e87a P 7201b2b1dba94a56929d3723f0acd0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } }
I20260620 18:32:58.731830 18860 ts_tablet_manager.cc:1428] T 94e3a0a5a56846ac83f9a618a760e87a P 7201b2b1dba94a56929d3723f0acd0a5: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 18:32:58.733824 18884 ts_tablet_manager.cc:1918] T b0de64a506394e17a1919054b4f6e125 P 74b98e19928d4abdbd1e062eddc64d68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:32:58.733955 18884 log.cc:1196] T b0de64a506394e17a1919054b4f6e125 P 74b98e19928d4abdbd1e062eddc64d68: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-1/wal/wals/b0de64a506394e17a1919054b4f6e125
I20260620 18:32:58.734352 18884 ts_tablet_manager.cc:1939] T b0de64a506394e17a1919054b4f6e125 P 74b98e19928d4abdbd1e062eddc64d68: Deleting consensus metadata
I20260620 18:32:58.735750 18336 catalog_manager.cc:4841] TS 74b98e19928d4abdbd1e062eddc64d68 (127.10.131.2:41595): tablet b0de64a506394e17a1919054b4f6e125 (table test-table [id=bea2f443d9574e9ea78e42f5391d64dd]) successfully deleted
I20260620 18:32:58.737067 18883 ts_tablet_manager.cc:1918] T b0de64a506394e17a1919054b4f6e125 P 58f75de98c4c4930b6ee00f9c2c1a33f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:32:58.737864 18883 log.cc:1196] T b0de64a506394e17a1919054b4f6e125 P 58f75de98c4c4930b6ee00f9c2c1a33f: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-2/wal/wals/b0de64a506394e17a1919054b4f6e125
I20260620 18:32:58.738389 18883 ts_tablet_manager.cc:1939] T b0de64a506394e17a1919054b4f6e125 P 58f75de98c4c4930b6ee00f9c2c1a33f: Deleting consensus metadata
I20260620 18:32:58.738655 18936 tablet_bootstrap.cc:492] T 94e3a0a5a56846ac83f9a618a760e87a P 58f75de98c4c4930b6ee00f9c2c1a33f: No bootstrap required, opened a new log
I20260620 18:32:58.738865 18936 ts_tablet_manager.cc:1397] T 94e3a0a5a56846ac83f9a618a760e87a P 58f75de98c4c4930b6ee00f9c2c1a33f: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260620 18:32:58.739545 18334 catalog_manager.cc:4841] TS 58f75de98c4c4930b6ee00f9c2c1a33f (127.10.131.3:42999): tablet b0de64a506394e17a1919054b4f6e125 (table test-table [id=bea2f443d9574e9ea78e42f5391d64dd]) successfully deleted
I20260620 18:32:58.739792 18936 raft_consensus.cc:348] T 94e3a0a5a56846ac83f9a618a760e87a P 58f75de98c4c4930b6ee00f9c2c1a33f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } }
I20260620 18:32:58.740026 18936 raft_consensus.cc:374] T 94e3a0a5a56846ac83f9a618a760e87a P 58f75de98c4c4930b6ee00f9c2c1a33f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:58.740120 18936 raft_consensus.cc:729] T 94e3a0a5a56846ac83f9a618a760e87a P 58f75de98c4c4930b6ee00f9c2c1a33f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f75de98c4c4930b6ee00f9c2c1a33f, State: Initialized, Role: FOLLOWER
I20260620 18:32:58.740403 18936 consensus_queue.cc:260] T 94e3a0a5a56846ac83f9a618a760e87a P 58f75de98c4c4930b6ee00f9c2c1a33f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } }
I20260620 18:32:58.741189 18935 tablet_bootstrap.cc:492] T 94e3a0a5a56846ac83f9a618a760e87a P 74b98e19928d4abdbd1e062eddc64d68: No bootstrap required, opened a new log
I20260620 18:32:58.741420 18935 ts_tablet_manager.cc:1397] T 94e3a0a5a56846ac83f9a618a760e87a P 74b98e19928d4abdbd1e062eddc64d68: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.003s
I20260620 18:32:58.741421 18936 ts_tablet_manager.cc:1428] T 94e3a0a5a56846ac83f9a618a760e87a P 58f75de98c4c4930b6ee00f9c2c1a33f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 18:32:58.743331 18935 raft_consensus.cc:348] T 94e3a0a5a56846ac83f9a618a760e87a P 74b98e19928d4abdbd1e062eddc64d68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } }
I20260620 18:32:58.743606 18935 raft_consensus.cc:374] T 94e3a0a5a56846ac83f9a618a760e87a P 74b98e19928d4abdbd1e062eddc64d68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:32:58.743703 18935 raft_consensus.cc:729] T 94e3a0a5a56846ac83f9a618a760e87a P 74b98e19928d4abdbd1e062eddc64d68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74b98e19928d4abdbd1e062eddc64d68, State: Initialized, Role: FOLLOWER
I20260620 18:32:58.743943 18935 consensus_queue.cc:260] T 94e3a0a5a56846ac83f9a618a760e87a P 74b98e19928d4abdbd1e062eddc64d68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } }
I20260620 18:32:58.745458 18935 ts_tablet_manager.cc:1428] T 94e3a0a5a56846ac83f9a618a760e87a P 74b98e19928d4abdbd1e062eddc64d68: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260620 18:32:58.769727 18891 raft_consensus.cc:482] T 94e3a0a5a56846ac83f9a618a760e87a P 58f75de98c4c4930b6ee00f9c2c1a33f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:32:58.769959 18891 raft_consensus.cc:504] T 94e3a0a5a56846ac83f9a618a760e87a P 58f75de98c4c4930b6ee00f9c2c1a33f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } }
I20260620 18:32:58.770617 18891 leader_election.cc:288] T 94e3a0a5a56846ac83f9a618a760e87a P 58f75de98c4c4930b6ee00f9c2c1a33f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7201b2b1dba94a56929d3723f0acd0a5 (127.10.131.1:35645), 74b98e19928d4abdbd1e062eddc64d68 (127.10.131.2:41595)
I20260620 18:32:58.771148 18650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e3a0a5a56846ac83f9a618a760e87a" candidate_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74b98e19928d4abdbd1e062eddc64d68" is_pre_election: true
I20260620 18:32:58.771170 18511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e3a0a5a56846ac83f9a618a760e87a" candidate_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7201b2b1dba94a56929d3723f0acd0a5" is_pre_election: true
I20260620 18:32:58.771384 18650 raft_consensus.cc:2454] T 94e3a0a5a56846ac83f9a618a760e87a P 74b98e19928d4abdbd1e062eddc64d68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f75de98c4c4930b6ee00f9c2c1a33f in term 0.
I20260620 18:32:58.771445 18511 raft_consensus.cc:2454] T 94e3a0a5a56846ac83f9a618a760e87a P 7201b2b1dba94a56929d3723f0acd0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f75de98c4c4930b6ee00f9c2c1a33f in term 0.
I20260620 18:32:58.771889 18718 leader_election.cc:302] T 94e3a0a5a56846ac83f9a618a760e87a P 58f75de98c4c4930b6ee00f9c2c1a33f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f75de98c4c4930b6ee00f9c2c1a33f, 74b98e19928d4abdbd1e062eddc64d68; no voters:
I20260620 18:32:58.772187 18891 raft_consensus.cc:2781] T 94e3a0a5a56846ac83f9a618a760e87a P 58f75de98c4c4930b6ee00f9c2c1a33f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:32:58.772338 18891 raft_consensus.cc:482] T 94e3a0a5a56846ac83f9a618a760e87a P 58f75de98c4c4930b6ee00f9c2c1a33f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:32:58.772471 18891 raft_consensus.cc:3037] T 94e3a0a5a56846ac83f9a618a760e87a P 58f75de98c4c4930b6ee00f9c2c1a33f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:58.774742 18891 raft_consensus.cc:504] T 94e3a0a5a56846ac83f9a618a760e87a P 58f75de98c4c4930b6ee00f9c2c1a33f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } }
I20260620 18:32:58.775485 18891 leader_election.cc:288] T 94e3a0a5a56846ac83f9a618a760e87a P 58f75de98c4c4930b6ee00f9c2c1a33f [CANDIDATE]: Term 1 election: Requested vote from peers 7201b2b1dba94a56929d3723f0acd0a5 (127.10.131.1:35645), 74b98e19928d4abdbd1e062eddc64d68 (127.10.131.2:41595)
I20260620 18:32:58.775782 18650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e3a0a5a56846ac83f9a618a760e87a" candidate_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74b98e19928d4abdbd1e062eddc64d68"
I20260620 18:32:58.775823 18511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e3a0a5a56846ac83f9a618a760e87a" candidate_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7201b2b1dba94a56929d3723f0acd0a5"
I20260620 18:32:58.776019 18511 raft_consensus.cc:3037] T 94e3a0a5a56846ac83f9a618a760e87a P 7201b2b1dba94a56929d3723f0acd0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:58.776047 18650 raft_consensus.cc:3037] T 94e3a0a5a56846ac83f9a618a760e87a P 74b98e19928d4abdbd1e062eddc64d68 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:32:58.778017 18650 raft_consensus.cc:2454] T 94e3a0a5a56846ac83f9a618a760e87a P 74b98e19928d4abdbd1e062eddc64d68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f75de98c4c4930b6ee00f9c2c1a33f in term 1.
I20260620 18:32:58.778497 18718 leader_election.cc:302] T 94e3a0a5a56846ac83f9a618a760e87a P 58f75de98c4c4930b6ee00f9c2c1a33f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f75de98c4c4930b6ee00f9c2c1a33f, 74b98e19928d4abdbd1e062eddc64d68; no voters:
I20260620 18:32:58.778823 18891 raft_consensus.cc:2781] T 94e3a0a5a56846ac83f9a618a760e87a P 58f75de98c4c4930b6ee00f9c2c1a33f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:32:58.779050 18511 raft_consensus.cc:2454] T 94e3a0a5a56846ac83f9a618a760e87a P 7201b2b1dba94a56929d3723f0acd0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f75de98c4c4930b6ee00f9c2c1a33f in term 1.
I20260620 18:32:58.779451 18891 raft_consensus.cc:686] T 94e3a0a5a56846ac83f9a618a760e87a P 58f75de98c4c4930b6ee00f9c2c1a33f [term 1 LEADER]: Becoming Leader. State: Replica: 58f75de98c4c4930b6ee00f9c2c1a33f, State: Running, Role: LEADER
I20260620 18:32:58.779707 18891 consensus_queue.cc:237] T 94e3a0a5a56846ac83f9a618a760e87a P 58f75de98c4c4930b6ee00f9c2c1a33f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } }
I20260620 18:32:58.782523 18349 catalog_manager.cc:5495] T 94e3a0a5a56846ac83f9a618a760e87a P 58f75de98c4c4930b6ee00f9c2c1a33f reported cstate change: term changed from 0 to 1, leader changed from <none> to 58f75de98c4c4930b6ee00f9c2c1a33f (127.10.131.3). New cstate: current_term: 1 leader_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f75de98c4c4930b6ee00f9c2c1a33f" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42999 } health_report { overall_health: HEALTHY } } }
I20260620 18:32:58.821861 18953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:58.811651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40422 (local address 127.10.131.3:42999)
0620 18:32:58.816798 (+ 5147us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:58.816807 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:58.816832 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:32:58.816934 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:58.816942 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:58.817045 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:58.817626 (+ 581us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:32:58.817824 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:58.818551 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:58.819975 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:58.820423 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:58.820576 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:58.821079 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:32:58.821224 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:32:58.821327 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:58.821470 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 18:32:58.821542 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4955,"thread_start_us":4798,"threads_started":1}
I20260620 18:32:58.831842 18511 raft_consensus.cc:1261] T 94e3a0a5a56846ac83f9a618a760e87a P 7201b2b1dba94a56929d3723f0acd0a5 [term 1 FOLLOWER]: Refusing update from remote peer 58f75de98c4c4930b6ee00f9c2c1a33f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:58.831836 18650 raft_consensus.cc:1261] T 94e3a0a5a56846ac83f9a618a760e87a P 74b98e19928d4abdbd1e062eddc64d68 [term 1 FOLLOWER]: Refusing update from remote peer 58f75de98c4c4930b6ee00f9c2c1a33f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:32:58.832456 18920 consensus_queue.cc:1035] T 94e3a0a5a56846ac83f9a618a760e87a P 58f75de98c4c4930b6ee00f9c2c1a33f [LEADER]: Connected to new peer: Peer: permanent_uuid: "74b98e19928d4abdbd1e062eddc64d68" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:58.832970 18891 consensus_queue.cc:1035] T 94e3a0a5a56846ac83f9a618a760e87a P 58f75de98c4c4930b6ee00f9c2c1a33f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201b2b1dba94a56929d3723f0acd0a5" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:32:58.882076 18347 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:32:58.882277 18347 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:32:58.884954 18347 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 012beec4185d4c8484308e45c3301e34: Sending DeleteTablet for 3 replicas of tablet 94e3a0a5a56846ac83f9a618a760e87a
I20260620 18:32:58.885769 18486 tablet_service.cc:1514] Processing DeleteTablet for tablet 94e3a0a5a56846ac83f9a618a760e87a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:58 UTC) from {username='oryx', principal='oryx/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:44582
I20260620 18:32:58.885783 18625 tablet_service.cc:1514] Processing DeleteTablet for tablet 94e3a0a5a56846ac83f9a618a760e87a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:58 UTC) from {username='oryx', principal='oryx/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:42922
I20260620 18:32:58.885831 18764 tablet_service.cc:1514] Processing DeleteTablet for tablet 94e3a0a5a56846ac83f9a618a760e87a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:32:58 UTC) from {username='oryx', principal='oryx/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:56828
I20260620 18:32:58.886058 18884 tablet_replica.cc:330] T 94e3a0a5a56846ac83f9a618a760e87a P 74b98e19928d4abdbd1e062eddc64d68: stopping tablet replica
I20260620 18:32:58.886089 18883 tablet_replica.cc:330] T 94e3a0a5a56846ac83f9a618a760e87a P 58f75de98c4c4930b6ee00f9c2c1a33f: stopping tablet replica
I20260620 18:32:58.886214 18885 tablet_replica.cc:330] T 94e3a0a5a56846ac83f9a618a760e87a P 7201b2b1dba94a56929d3723f0acd0a5: stopping tablet replica
I20260620 18:32:58.886262 18884 raft_consensus.cc:2229] T 94e3a0a5a56846ac83f9a618a760e87a P 74b98e19928d4abdbd1e062eddc64d68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:32:58.886283 18883 raft_consensus.cc:2229] T 94e3a0a5a56846ac83f9a618a760e87a P 58f75de98c4c4930b6ee00f9c2c1a33f [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:32:58.886487 18884 raft_consensus.cc:2258] T 94e3a0a5a56846ac83f9a618a760e87a P 74b98e19928d4abdbd1e062eddc64d68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:58.886453 18885 raft_consensus.cc:2229] T 94e3a0a5a56846ac83f9a618a760e87a P 7201b2b1dba94a56929d3723f0acd0a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:32:58.886636 18883 raft_consensus.cc:2258] T 94e3a0a5a56846ac83f9a618a760e87a P 58f75de98c4c4930b6ee00f9c2c1a33f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:58.886705 18885 raft_consensus.cc:2258] T 94e3a0a5a56846ac83f9a618a760e87a P 7201b2b1dba94a56929d3723f0acd0a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:32:58.887873 18884 ts_tablet_manager.cc:1905] T 94e3a0a5a56846ac83f9a618a760e87a P 74b98e19928d4abdbd1e062eddc64d68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:58.887879 18883 ts_tablet_manager.cc:1905] T 94e3a0a5a56846ac83f9a618a760e87a P 58f75de98c4c4930b6ee00f9c2c1a33f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:58.887974 18885 ts_tablet_manager.cc:1905] T 94e3a0a5a56846ac83f9a618a760e87a P 7201b2b1dba94a56929d3723f0acd0a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:32:58.889038 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 18420
I20260620 18:32:58.895206 18885 ts_tablet_manager.cc:1918] T 94e3a0a5a56846ac83f9a618a760e87a P 7201b2b1dba94a56929d3723f0acd0a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:32:58.895198 18884 ts_tablet_manager.cc:1918] T 94e3a0a5a56846ac83f9a618a760e87a P 74b98e19928d4abdbd1e062eddc64d68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:32:58.895354 18884 log.cc:1196] T 94e3a0a5a56846ac83f9a618a760e87a P 74b98e19928d4abdbd1e062eddc64d68: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-1/wal/wals/94e3a0a5a56846ac83f9a618a760e87a
I20260620 18:32:58.895490 18885 log.cc:1196] T 94e3a0a5a56846ac83f9a618a760e87a P 7201b2b1dba94a56929d3723f0acd0a5: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-0/wal/wals/94e3a0a5a56846ac83f9a618a760e87a
I20260620 18:32:58.895887 18884 ts_tablet_manager.cc:1939] T 94e3a0a5a56846ac83f9a618a760e87a P 74b98e19928d4abdbd1e062eddc64d68: Deleting consensus metadata
I20260620 18:32:58.895936 18885 ts_tablet_manager.cc:1939] T 94e3a0a5a56846ac83f9a618a760e87a P 7201b2b1dba94a56929d3723f0acd0a5: Deleting consensus metadata
I20260620 18:32:58.896426 18883 ts_tablet_manager.cc:1918] T 94e3a0a5a56846ac83f9a618a760e87a P 58f75de98c4c4930b6ee00f9c2c1a33f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:32:58.896693 18883 log.cc:1196] T 94e3a0a5a56846ac83f9a618a760e87a P 58f75de98c4c4930b6ee00f9c2c1a33f: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781980314095189-10764-0/minicluster-data/ts-2/wal/wals/94e3a0a5a56846ac83f9a618a760e87a
I20260620 18:32:58.897022 18336 catalog_manager.cc:4841] TS 74b98e19928d4abdbd1e062eddc64d68 (127.10.131.2:41595): tablet 94e3a0a5a56846ac83f9a618a760e87a (table test-table [id=cb7165c353bf4b9d80b9077bbe9e6400]) successfully deleted
I20260620 18:32:58.897087 18334 catalog_manager.cc:4841] TS 7201b2b1dba94a56929d3723f0acd0a5 (127.10.131.1:35645): tablet 94e3a0a5a56846ac83f9a618a760e87a (table test-table [id=cb7165c353bf4b9d80b9077bbe9e6400]) successfully deleted
I20260620 18:32:58.897162 18883 ts_tablet_manager.cc:1939] T 94e3a0a5a56846ac83f9a618a760e87a P 58f75de98c4c4930b6ee00f9c2c1a33f: Deleting consensus metadata
I20260620 18:32:58.897993 18334 catalog_manager.cc:4841] TS 58f75de98c4c4930b6ee00f9c2c1a33f (127.10.131.3:42999): tablet 94e3a0a5a56846ac83f9a618a760e87a (table test-table [id=cb7165c353bf4b9d80b9077bbe9e6400]) successfully deleted
Jun 20 18:32:58 dist-test-slave-cbn3 krb5kdc[18304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.131.1@KRBTEST.COM
I20260620 18:32:58.912360 18942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:58.890059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35086 (local address 127.10.131.1:35645)
0620 18:32:58.890219 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:58.890229 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:58.897474 (+ 7245us) server_negotiation.cc:407] Connection header received
0620 18:32:58.897704 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:58.897715 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:58.897775 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:58.897896 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:58.899352 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:58.900203 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:58.901594 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:58.902026 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:58.902173 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:58.906270 (+ 4097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:58.906285 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:58.906301 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:58.906335 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:58.909293 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:58.909991 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:58.909999 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:58.910001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:58.910125 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:58.910576 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:58.910582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:58.910585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:58.911046 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:58.911340 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:58.911906 (+ 566us) server_negotiation.cc:299] Negotiation successful
0620 18:32:58.912068 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 18:32:58.913034 18551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:59.042102 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 18420
I20260620 18:32:59.064358 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 18560
Jun 20 18:32:59 dist-test-slave-cbn3 krb5kdc[18304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.131.2@KRBTEST.COM
I20260620 18:32:59.078496 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:59.065052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57906 (local address 127.10.131.2:41595)
0620 18:32:59.065540 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:59.065544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:59.065558 (+ 14us) server_negotiation.cc:407] Connection header received
0620 18:32:59.065607 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:59.065610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:59.065667 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:59.065802 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:59.066999 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:59.067750 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:59.069026 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:59.069405 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:59.069495 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:59.073361 (+ 3866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:59.073372 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:59.073386 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:59.073421 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:59.076180 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:59.076743 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:59.076749 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:59.076751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:59.076924 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:59.077234 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:59.077237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:59.077239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:59.077576 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:59.077782 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:59.077972 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 18:32:59.078123 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":250,"threads_started":1}
I20260620 18:32:59.079101 18690 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:59.198307 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 18560
I20260620 18:32:59.219511 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 18699
Jun 20 18:32:59 dist-test-slave-cbn3 krb5kdc[18304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.131.3@KRBTEST.COM
I20260620 18:32:59.234189 18953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:32:59.220245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40430 (local address 127.10.131.3:42999)
0620 18:32:59.220419 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 18:32:59.220424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:32:59.220492 (+ 68us) server_negotiation.cc:407] Connection header received
0620 18:32:59.220638 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:32:59.220642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:32:59.220716 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:32:59.220803 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:32:59.222286 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:59.222975 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:59.224173 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:32:59.224544 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:32:59.224633 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:32:59.229002 (+ 4369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:32:59.229017 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:32:59.229034 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:32:59.229067 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:32:59.231614 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:59.232226 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:59.232232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:59.232234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:59.232349 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:32:59.232724 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:32:59.232732 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:32:59.232734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:32:59.233121 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:32:59.233347 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 18:32:59.233599 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 18:32:59.233808 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:32:59.234856 18829 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:59.354341 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 18699
I20260620 18:32:59.375526 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 18317
I20260620 18:32:59.376649 18389 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:32:59.491127 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 18317
2026-06-20T18:32:59Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6414 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:32:59 dist-test-slave-cbn3 krb5kdc[18968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:32:59 dist-test-slave-cbn3 krb5kdc[18968](info): set up 2 sockets
Jun 20 18:32:59 dist-test-slave-cbn3 krb5kdc[18968](info): commencing operation
krb5kdc: starting...
W20260620 18:33:00.589888 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.007s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:33:00 dist-test-slave-cbn3 krb5kdc[18968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:33:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:33:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.131.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
I20260620 18:33:00.806038 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:41785
--webserver_interface=127.10.131.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:42859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:41785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:33:01.123186 18981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:33:01.123462 18981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:33:01.123548 18981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:33:01.130729 18981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:33:01.130826 18981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:33:01.130868 18981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:33:01.130921 18981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:33:01.130967 18981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:33:01.140581 18981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:42859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:41785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:41785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:33:01.141808 18981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:33:01.143002 18981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:33:01.154305 18987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:01.154335 18986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:01.154428 18989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:33:01.154999 18981 server_base.cc:1034] running on GCE node
I20260620 18:33:01.155939 18981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:33:01.157408 18981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:33:01.158645 18981 hybrid_clock.cc:648] HybridClock initialized: now 1781980381158606 us; error 46 us; skew 500 ppm
I20260620 18:33:01.158979 18981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T18:33:01Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1749 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:33:01 dist-test-slave-cbn3 krb5kdc[18995](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:33:01 dist-test-slave-cbn3 krb5kdc[18995](info): set up 2 sockets
Jun 20 18:33:01 dist-test-slave-cbn3 krb5kdc[18995](info): commencing operation
krb5kdc: starting...
W20260620 18:33:02.330056 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:33:02 dist-test-slave-cbn3 krb5kdc[18995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:33:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:33:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.131.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
I20260620 18:33:02.548938 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:35671
--webserver_interface=127.10.131.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:36025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:35671
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:33:02.878041 19008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:33:02.878326 19008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:33:02.878425 19008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:33:02.886085 19008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:33:02.886189 19008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:33:02.886234 19008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:33:02.886289 19008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:33:02.886332 19008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:33:02.895967 19008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:36025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:35671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:35671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:33:02.897315 19008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:33:02.898586 19008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:33:02.910310 19016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:02.910310 19014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:02.910382 19013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:33:02.914355 19008 server_base.cc:1034] running on GCE node
I20260620 18:33:02.915257 19008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:33:02.916730 19008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:33:02.917981 19008 hybrid_clock.cc:648] HybridClock initialized: now 1781980382917915 us; error 64 us; skew 500 ppm
I20260620 18:33:02.918346 19008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:33:02 dist-test-slave-cbn3 krb5kdc[18995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980382, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:33:02.925261 19008 init.cc:377] Logged in from keytab as kudu/127.10.131.62@KRBTEST.COM (short username kudu)
I20260620 18:33:02.926904 19008 webserver.cc:460] Webserver started at http://127.10.131.62:35237/ using document root <none> and password file <none>
I20260620 18:33:02.927659 19008 fs_manager.cc:359] Metadata directory not provided
I20260620 18:33:02.927806 19008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:33:02.928122 19008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:33:02.931181 19008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "c7407a912a9b44a48ac5ad1fe746d21f"
format_stamp: "Formatted at 2026-06-20 18:33:02 on dist-test-slave-cbn3"
server_key: "21f543ff36d741b9da84346ecfab4756"
server_key_iv: "87b107037ee8372261b4269f7fe187e6"
server_key_version: "encryptionkey@0"
I20260620 18:33:02.931947 19008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "c7407a912a9b44a48ac5ad1fe746d21f"
format_stamp: "Formatted at 2026-06-20 18:33:02 on dist-test-slave-cbn3"
server_key: "21f543ff36d741b9da84346ecfab4756"
server_key_iv: "87b107037ee8372261b4269f7fe187e6"
server_key_version: "encryptionkey@0"
I20260620 18:33:02.937141 19008 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:33:02.940537 19023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:02.941895 19008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:33:02.942075 19008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "c7407a912a9b44a48ac5ad1fe746d21f"
format_stamp: "Formatted at 2026-06-20 18:33:02 on dist-test-slave-cbn3"
server_key: "21f543ff36d741b9da84346ecfab4756"
server_key_iv: "87b107037ee8372261b4269f7fe187e6"
server_key_version: "encryptionkey@0"
I20260620 18:33:02.942269 19008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:33:02.964479 19008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:33:02.969089 19008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:33:02.969386 19008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:33:02.987728 19008 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:35671
I20260620 18:33:02.987748 19074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:35671 every 8 connection(s)
I20260620 18:33:02.989198 19008 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:33:02.991456 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 19008
I20260620 18:33:02.991676 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:33:02.992295 10764 external_mini_cluster.cc:1428] Setting key 0bdf69d51cfd6b93f0ae1e44e5816d7c
I20260620 18:33:02.993582 19075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:33:03 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:33:03.008260 19075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f: Bootstrap starting.
I20260620 18:33:03.012909 19075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f: Neither blocks nor log segments found. Creating new log.
I20260620 18:33:03.014081 19075 log.cc:824] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:33:03.016872 19078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:02.994234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54642 (local address 127.10.131.62:35671)
0620 18:33:02.995014 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:02.995030 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:02.995080 (+ 50us) server_negotiation.cc:407] Connection header received
0620 18:33:02.997223 (+ 2143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:02.997250 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:02.997791 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:02.998246 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:02.999546 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:03.001726 (+ 2180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:03.002951 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:03.004011 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:03.004202 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:03.008248 (+ 4046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:03.008260 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:03.008282 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:33:03.008343 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:03.012396 (+ 4053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:03.013191 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:03.013198 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:03.013204 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:03.013389 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:03.013768 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:03.013773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:03.013776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:03.014353 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:03.014638 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:03.015106 (+ 468us) server_negotiation.cc:299] Negotiation successful
0620 18:33:03.015577 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":309,"threads_started":1}
I20260620 18:33:03.017486 19075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f: No bootstrap required, opened a new log
I20260620 18:33:03.025022 19075 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7407a912a9b44a48ac5ad1fe746d21f" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35671 } }
I20260620 18:33:03.025415 19075 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:33:03.025535 19075 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7407a912a9b44a48ac5ad1fe746d21f, State: Initialized, Role: FOLLOWER
I20260620 18:33:03.026182 19075 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7407a912a9b44a48ac5ad1fe746d21f" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35671 } }
I20260620 18:33:03.026472 19075 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:33:03.026624 19075 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:33:03.026794 19075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:33:03.028985 19075 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7407a912a9b44a48ac5ad1fe746d21f" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35671 } }
I20260620 18:33:03.029556 19075 leader_election.cc:302] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7407a912a9b44a48ac5ad1fe746d21f; no voters:
I20260620 18:33:03.030246 19075 leader_election.cc:288] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:33:03.030386 19080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:33:03.031481 19080 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f [term 1 LEADER]: Becoming Leader. State: Replica: c7407a912a9b44a48ac5ad1fe746d21f, State: Running, Role: LEADER
I20260620 18:33:03.031992 19080 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7407a912a9b44a48ac5ad1fe746d21f" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35671 } }
I20260620 18:33:03.032356 19075 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:33:03.034938 19081 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7407a912a9b44a48ac5ad1fe746d21f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7407a912a9b44a48ac5ad1fe746d21f" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35671 } } }
I20260620 18:33:03.034883 19082 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7407a912a9b44a48ac5ad1fe746d21f. Latest consensus state: current_term: 1 leader_uuid: "c7407a912a9b44a48ac5ad1fe746d21f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7407a912a9b44a48ac5ad1fe746d21f" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35671 } } }
I20260620 18:33:03.035347 19081 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f [sys.catalog]: This master's current role is: LEADER
I20260620 18:33:03.035379 19082 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f [sys.catalog]: This master's current role is: LEADER
I20260620 18:33:03.037556 19089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:33:03.043126 19089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:33:03.052373 19089 catalog_manager.cc:1347] Generated new cluster ID: d49bfd83e27c40c3a874c1ff34cb7565
I20260620 18:33:03.052506 19089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:33:03.059902 19089 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:33:03.061780 19089 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:33:03.074767 19089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f: Generated new TSK 0
I20260620 18:33:03.075644 19089 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.131.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab.
Entry for principal kudu/127.10.131.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab.
I20260620 18:33:03.132059 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.1:0
--local_ip_for_outbound_sockets=127.10.131.1
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:35671
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:36025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:33:03.453732 19101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:33:03.454015 19101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:33:03.454182 19101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:33:03.461531 19101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:33:03.461637 19101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:33:03.461853 19101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.1
I20260620 18:33:03.471531 19101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:36025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:35671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.1
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:33:03.472786 19101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:33:03.474112 19101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:33:03.488390 19106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:03.488449 19107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:03.488449 19109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:33:03.489262 19101 server_base.cc:1034] running on GCE node
I20260620 18:33:03.489897 19101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:33:03.490785 19101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:33:03.491981 19101 hybrid_clock.cc:648] HybridClock initialized: now 1781980383491936 us; error 36 us; skew 500 ppm
I20260620 18:33:03.492306 19101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:33:03 dist-test-slave-cbn3 krb5kdc[18995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980383, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:33:03.495604 19101 init.cc:377] Logged in from keytab as kudu/127.10.131.1@KRBTEST.COM (short username kudu)
I20260620 18:33:03.497124 19101 webserver.cc:460] Webserver started at http://127.10.131.1:33937/ using document root <none> and password file <none>
I20260620 18:33:03.497821 19101 fs_manager.cc:359] Metadata directory not provided
I20260620 18:33:03.497963 19101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:33:03.498270 19101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:33:03.501196 19101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-0/data/instance:
uuid: "13c284cb03f44a0184b459a0d8feb06d"
format_stamp: "Formatted at 2026-06-20 18:33:03 on dist-test-slave-cbn3"
server_key: "f7730063a6be2db8f94ab44c0347d172"
server_key_iv: "14849ef3d41bf661f3c73571a73cc411"
server_key_version: "encryptionkey@0"
I20260620 18:33:03.502009 19101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance:
uuid: "13c284cb03f44a0184b459a0d8feb06d"
format_stamp: "Formatted at 2026-06-20 18:33:03 on dist-test-slave-cbn3"
server_key: "f7730063a6be2db8f94ab44c0347d172"
server_key_iv: "14849ef3d41bf661f3c73571a73cc411"
server_key_version: "encryptionkey@0"
I20260620 18:33:03.507589 19101 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:33:03.511399 19116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:03.512732 19101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:33:03.512918 19101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-0/wal
uuid: "13c284cb03f44a0184b459a0d8feb06d"
format_stamp: "Formatted at 2026-06-20 18:33:03 on dist-test-slave-cbn3"
server_key: "f7730063a6be2db8f94ab44c0347d172"
server_key_iv: "14849ef3d41bf661f3c73571a73cc411"
server_key_version: "encryptionkey@0"
I20260620 18:33:03.513123 19101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:33:03.528520 19101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:33:03.532995 19101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:33:03.533280 19101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:33:03.534351 19101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:33:03.535619 19101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:33:03.535696 19101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:03.535838 19101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:33:03.535903 19101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:03.568326 19101 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.1:37579
I20260620 18:33:03.568373 19228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.1:37579 every 8 connection(s)
I20260620 18:33:03.569701 19101 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
I20260620 18:33:03.573038 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 19101
I20260620 18:33:03.573225 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance
I20260620 18:33:03.573945 10764 external_mini_cluster.cc:1428] Setting key dd592a498c940792d3609e66296dfb58
Jun 20 18:33:03 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980383, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:33:03.590613 19231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:03.572119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:54185 (local address 127.10.131.62:35671)
0620 18:33:03.572588 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:03.572594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:03.573649 (+ 1055us) server_negotiation.cc:407] Connection header received
0620 18:33:03.576048 (+ 2399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:03.576054 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:03.576130 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:03.576245 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:03.577816 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:03.578919 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:03.580424 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:03.581179 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:03.581317 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:03.585158 (+ 3841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:03.585164 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:03.585167 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:33:03.585199 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:03.587612 (+ 2413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:03.588316 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:03.588322 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:03.588325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:03.588449 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:03.588894 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:03.588899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:03.588902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:03.589171 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:03.589328 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:03.590034 (+ 706us) server_negotiation.cc:299] Negotiation successful
0620 18:33:03.590252 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":220,"threads_started":1}
I20260620 18:33:03.591709 19232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:03.572649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.1:54185)
0620 18:33:03.573458 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:03.573526 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:03.575808 (+ 2282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:03.576428 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:03.576435 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:03.577112 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:03.577655 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:03.577672 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:03.579073 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:03.579076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:03.580286 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:03.580297 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:03.581371 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:03.581376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:03.581578 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:03.582253 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:03.582275 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:03.584923 (+ 2648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:33:03.587754 (+ 2831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:33:03.587763 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:33:03.587780 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:33:03.588193 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:33:03.588557 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:33:03.588562 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:33:03.588566 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:33:03.588757 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:33:03.589298 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:33:03.589306 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:33:03.589660 (+ 354us) client_negotiation.cc:769] Sending connection context
0620 18:33:03.589922 (+ 262us) client_negotiation.cc:240] Negotiation successful
0620 18:33:03.590466 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":353,"threads_started":1}
I20260620 18:33:03.593353 19229 heartbeater.cc:347] Connected to a master server at 127.10.131.62:35671
I20260620 18:33:03.593673 19229 heartbeater.cc:464] Registering TS with master...
I20260620 18:33:03.594390 19229 heartbeater.cc:511] Master 127.10.131.62:35671 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.131.2@KRBTEST.COM; defaulting to no policy
I20260620 18:33:03.596529 19040 ts_manager.cc:194] Registered new tserver with Master: 13c284cb03f44a0184b459a0d8feb06d (127.10.131.1:37579)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.2@KRBTEST.COM" created.
I20260620 18:33:03.598227 19040 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.1@KRBTEST.COM'} at 127.10.131.1:54185
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab.
Entry for principal kudu/127.10.131.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab.
I20260620 18:33:03.626081 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.2:0
--local_ip_for_outbound_sockets=127.10.131.2
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:35671
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:36025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:33:03.947585 19235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:33:03.947908 19235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:33:03.948084 19235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:33:03.955415 19235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:33:03.955574 19235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:33:03.955762 19235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.2
I20260620 18:33:03.965667 19235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:36025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:35671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.2
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:33:03.967037 19235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:33:03.968345 19235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:33:03.981567 19240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:03.981601 19243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:03.981567 19241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:33:03.982713 19235 server_base.cc:1034] running on GCE node
I20260620 18:33:03.983311 19235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:33:03.984262 19235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:33:03.985491 19235 hybrid_clock.cc:648] HybridClock initialized: now 1781980383985435 us; error 43 us; skew 500 ppm
I20260620 18:33:03.985809 19235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:33:03 dist-test-slave-cbn3 krb5kdc[18995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980383, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:33:03.989233 19235 init.cc:377] Logged in from keytab as kudu/127.10.131.2@KRBTEST.COM (short username kudu)
I20260620 18:33:03.990875 19235 webserver.cc:460] Webserver started at http://127.10.131.2:34359/ using document root <none> and password file <none>
I20260620 18:33:03.991568 19235 fs_manager.cc:359] Metadata directory not provided
I20260620 18:33:03.991714 19235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:33:03.992022 19235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:33:03.994966 19235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-1/data/instance:
uuid: "bccbcc551de749f0955340b0a734d512"
format_stamp: "Formatted at 2026-06-20 18:33:03 on dist-test-slave-cbn3"
server_key: "28bc36fa85f584a946504d5b0f565590"
server_key_iv: "c034ba5ac770e048096673c5706cc59f"
server_key_version: "encryptionkey@0"
I20260620 18:33:03.995761 19235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance:
uuid: "bccbcc551de749f0955340b0a734d512"
format_stamp: "Formatted at 2026-06-20 18:33:03 on dist-test-slave-cbn3"
server_key: "28bc36fa85f584a946504d5b0f565590"
server_key_iv: "c034ba5ac770e048096673c5706cc59f"
server_key_version: "encryptionkey@0"
I20260620 18:33:04.000973 19235 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 18:33:04.004362 19250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:04.005769 19235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:33:04.005954 19235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-1/wal
uuid: "bccbcc551de749f0955340b0a734d512"
format_stamp: "Formatted at 2026-06-20 18:33:03 on dist-test-slave-cbn3"
server_key: "28bc36fa85f584a946504d5b0f565590"
server_key_iv: "c034ba5ac770e048096673c5706cc59f"
server_key_version: "encryptionkey@0"
I20260620 18:33:04.006151 19235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:33:04.019308 19235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:33:04.023763 19235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:33:04.024073 19235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:33:04.025104 19235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:33:04.026392 19235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:33:04.026507 19235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:04.026656 19235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:33:04.026732 19235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:04.059824 19235 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.2:35765
I20260620 18:33:04.059871 19362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.2:35765 every 8 connection(s)
I20260620 18:33:04.061175 19235 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
I20260620 18:33:04.066715 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 19235
I20260620 18:33:04.066890 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance
I20260620 18:33:04.067461 10764 external_mini_cluster.cc:1428] Setting key 02961cd0afdfae836c7a6771257c7fba
Jun 20 18:33:04 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980383, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:33:04.084726 19231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:04.063679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:45607 (local address 127.10.131.62:35671)
0620 18:33:04.063845 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:04.063849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:04.064963 (+ 1114us) server_negotiation.cc:407] Connection header received
0620 18:33:04.066887 (+ 1924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:04.066895 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:04.066991 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:04.067084 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:04.068947 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:04.069932 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:04.072520 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:04.073000 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:04.073100 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:04.077566 (+ 4466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:04.077574 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:04.077579 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:33:04.077626 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:04.081234 (+ 3608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:04.082159 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:04.082167 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:04.082173 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:04.082340 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:04.082755 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:04.082762 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:04.082768 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:04.083159 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:04.083321 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:04.084016 (+ 695us) server_negotiation.cc:299] Negotiation successful
0620 18:33:04.084285 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:33:04.085472 19365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:04.064061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.2:45607)
0620 18:33:04.064745 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:04.064781 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:04.066561 (+ 1780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:04.067337 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:04.067348 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:04.068068 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:04.068750 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:04.068769 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:04.070089 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:04.070095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:04.072359 (+ 2264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:04.072377 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:04.073215 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:04.073224 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:04.073543 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:04.074271 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:04.074304 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:04.077298 (+ 2994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:33:04.081409 (+ 4111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:33:04.081416 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:33:04.081430 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:33:04.082012 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:33:04.082450 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:33:04.082453 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:33:04.082456 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:33:04.082638 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:33:04.083275 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:33:04.083295 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:33:04.083644 (+ 349us) client_negotiation.cc:769] Sending connection context
0620 18:33:04.083864 (+ 220us) client_negotiation.cc:240] Negotiation successful
0620 18:33:04.084266 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":274,"threads_started":1}
I20260620 18:33:04.087018 19363 heartbeater.cc:347] Connected to a master server at 127.10.131.62:35671
I20260620 18:33:04.087340 19363 heartbeater.cc:464] Registering TS with master...
I20260620 18:33:04.088161 19363 heartbeater.cc:511] Master 127.10.131.62:35671 requested a full tablet report, sending...
I20260620 18:33:04.089682 19040 ts_manager.cc:194] Registered new tserver with Master: bccbcc551de749f0955340b0a734d512 (127.10.131.2:35765)
I20260620 18:33:04.090466 19040 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.2@KRBTEST.COM'} at 127.10.131.2:45607
WARNING: no policy specified for kudu/127.10.131.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab.
Entry for principal kudu/127.10.131.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab.
I20260620 18:33:04.118240 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.3:0
--local_ip_for_outbound_sockets=127.10.131.3
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:35671
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:36025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:33:04.439914 19368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:33:04.440274 19368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:33:04.440438 19368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:33:04.447860 19368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:33:04.448005 19368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:33:04.448186 19368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.3
I20260620 18:33:04.457734 19368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:36025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:35671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.3
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:33:04.459095 19368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:33:04.460511 19368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:33:04.474074 19373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:04.473989 19374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:04.474051 19376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:33:04.475888 19368 server_base.cc:1034] running on GCE node
I20260620 18:33:04.476487 19368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:33:04.477443 19368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:33:04.478725 19368 hybrid_clock.cc:648] HybridClock initialized: now 1781980384478677 us; error 33 us; skew 500 ppm
I20260620 18:33:04.479110 19368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:33:04 dist-test-slave-cbn3 krb5kdc[18995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980384, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:33:04.482434 19368 init.cc:377] Logged in from keytab as kudu/127.10.131.3@KRBTEST.COM (short username kudu)
I20260620 18:33:04.484054 19368 webserver.cc:460] Webserver started at http://127.10.131.3:34805/ using document root <none> and password file <none>
I20260620 18:33:04.484803 19368 fs_manager.cc:359] Metadata directory not provided
I20260620 18:33:04.485020 19368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:33:04.485437 19368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:33:04.489648 19368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-2/data/instance:
uuid: "cc741f493db34965aa402be52638c2f5"
format_stamp: "Formatted at 2026-06-20 18:33:04 on dist-test-slave-cbn3"
server_key: "b82ef8e5028bcdaec3c28934a4940608"
server_key_iv: "0d247c1ab9b91482032d8002b8f107fa"
server_key_version: "encryptionkey@0"
I20260620 18:33:04.490476 19368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance:
uuid: "cc741f493db34965aa402be52638c2f5"
format_stamp: "Formatted at 2026-06-20 18:33:04 on dist-test-slave-cbn3"
server_key: "b82ef8e5028bcdaec3c28934a4940608"
server_key_iv: "0d247c1ab9b91482032d8002b8f107fa"
server_key_version: "encryptionkey@0"
I20260620 18:33:04.495935 19368 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:33:04.499666 19383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:04.501092 19368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:33:04.501271 19368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-2/wal
uuid: "cc741f493db34965aa402be52638c2f5"
format_stamp: "Formatted at 2026-06-20 18:33:04 on dist-test-slave-cbn3"
server_key: "b82ef8e5028bcdaec3c28934a4940608"
server_key_iv: "0d247c1ab9b91482032d8002b8f107fa"
server_key_version: "encryptionkey@0"
I20260620 18:33:04.501483 19368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:33:04.521816 19368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:33:04.525992 19368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:33:04.526341 19368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:33:04.527371 19368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:33:04.528832 19368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:33:04.529018 19368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:04.529176 19368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:33:04.529256 19368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:04.564306 19368 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.3:40125
I20260620 18:33:04.564344 19495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.3:40125 every 8 connection(s)
I20260620 18:33:04.565908 19368 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
I20260620 18:33:04.569463 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 19368
I20260620 18:33:04.569633 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance
I20260620 18:33:04.570156 10764 external_mini_cluster.cc:1428] Setting key 9204d2cf28a1e784e9e8a31e8ebe2c22
Jun 20 18:33:04 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980384, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:33:04.586046 19231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:04.568469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:35975 (local address 127.10.131.62:35671)
0620 18:33:04.568694 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:04.568704 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:04.569976 (+ 1272us) server_negotiation.cc:407] Connection header received
0620 18:33:04.572538 (+ 2562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:04.572543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:04.572625 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:04.572731 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:04.574376 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:04.575134 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:04.576583 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:04.577038 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:04.577166 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:04.580428 (+ 3262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:04.580432 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:04.580464 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:33:04.580499 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:04.582957 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:04.583792 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:04.583798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:04.583803 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:04.583980 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:04.584477 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:04.584482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:04.584485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:04.584790 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:04.584990 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:04.585627 (+ 637us) server_negotiation.cc:299] Negotiation successful
0620 18:33:04.585779 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:33:04.587036 19498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:04.568943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.3:35975)
0620 18:33:04.569796 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:04.569856 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:04.572245 (+ 2389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:04.573169 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:04.573177 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:04.573647 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:04.574217 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:04.574235 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:04.575259 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:04.575263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:04.576435 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:04.576445 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:04.577257 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:04.577264 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:04.577510 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:04.578145 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:04.578166 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:04.580233 (+ 2067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:33:04.583080 (+ 2847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:33:04.583086 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:33:04.583097 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:33:04.583663 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:33:04.584096 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:33:04.584120 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:33:04.584125 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:33:04.584368 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:33:04.584969 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:33:04.584977 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:33:04.585285 (+ 308us) client_negotiation.cc:769] Sending connection context
0620 18:33:04.585561 (+ 276us) client_negotiation.cc:240] Negotiation successful
0620 18:33:04.585959 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":335,"threads_started":1}
I20260620 18:33:04.588285 19496 heartbeater.cc:347] Connected to a master server at 127.10.131.62:35671
I20260620 18:33:04.588585 19496 heartbeater.cc:464] Registering TS with master...
I20260620 18:33:04.589331 19496 heartbeater.cc:511] Master 127.10.131.62:35671 requested a full tablet report, sending...
I20260620 18:33:04.590792 19040 ts_manager.cc:194] Registered new tserver with Master: cc741f493db34965aa402be52638c2f5 (127.10.131.3:40125)
I20260620 18:33:04.591820 19040 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.3@KRBTEST.COM'} at 127.10.131.3:35975
I20260620 18:33:04.597563 10764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:33:04.601121 19229 heartbeater.cc:503] Master 127.10.131.62:35671 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.10.131.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.131.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.131.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.1.keytab.
Entry for principal oryx/127.10.131.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.1.keytab.
WARNING: no policy specified for oryx/127.10.131.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.131.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.131.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.2.keytab.
Entry for principal oryx/127.10.131.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.2.keytab.
WARNING: no policy specified for oryx/127.10.131.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.131.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.131.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.3.keytab.
Entry for principal oryx/127.10.131.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.3.keytab.
I20260620 18:33:04.730001 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 19101
Jun 20 18:33:04 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.1@KRBTEST.COM
I20260620 18:33:04.747628 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:04.731280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43222 (local address 127.10.131.1:37579)
0620 18:33:04.731907 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:04.731915 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:04.731977 (+ 62us) server_negotiation.cc:407] Connection header received
0620 18:33:04.732119 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:04.732126 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:04.732341 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:04.732476 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:04.734144 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:04.735170 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:04.736662 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:04.737362 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:04.737462 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:04.741782 (+ 4320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:04.741794 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:04.741810 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:33:04.741849 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:04.744731 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:04.745455 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:04.745462 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:04.745464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:04.745613 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:04.746012 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:04.746015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:04.746017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:04.746335 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:04.746545 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:04.746993 (+ 448us) server_negotiation.cc:299] Negotiation successful
0620 18:33:04.747198 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":297,"threads_started":1}
I20260620 18:33:04.749078 19224 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:33:04.832043 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 19101
I20260620 18:33:04.848546 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 19235
Jun 20 18:33:04 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.2@KRBTEST.COM
I20260620 18:33:04.864279 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:04.849445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52768 (local address 127.10.131.2:35765)
0620 18:33:04.849916 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:04.849923 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:04.849948 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:33:04.850006 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:04.850012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:04.850215 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:04.850384 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:04.851605 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:04.852544 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:04.853820 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:04.854358 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:04.854501 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:04.858550 (+ 4049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:04.858562 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:04.858577 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:33:04.858613 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:04.861585 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:04.862208 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:04.862215 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:04.862217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:04.862335 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:04.862707 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:04.862712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:04.862714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:04.863076 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:04.863317 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:04.863571 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 18:33:04.863785 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":235,"threads_started":1}
I20260620 18:33:04.865592 19358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:33:04.946786 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 19235
I20260620 18:33:04.963140 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 19368
Jun 20 18:33:04 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.3@KRBTEST.COM
I20260620 18:33:04.979189 19510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:04.964072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36118 (local address 127.10.131.3:40125)
0620 18:33:04.964815 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:04.964822 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:04.964902 (+ 80us) server_negotiation.cc:407] Connection header received
0620 18:33:04.964964 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:04.964970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:04.965190 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:04.965312 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:04.966617 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:04.967601 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:04.968807 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:04.969501 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:04.969606 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:04.973853 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:04.973864 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:04.973879 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:33:04.973911 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:04.976687 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:04.977359 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:04.977366 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:04.977368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:04.977492 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:04.977838 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:04.977841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:04.977843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:04.978172 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:04.978401 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:04.978614 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 18:33:04.978808 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":407,"threads_started":1}
I20260620 18:33:04.980422 19491 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:33:05.066565 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 19368
I20260620 18:33:05.082816 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 19008
I20260620 18:33:05.084216 19070 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:33:05.184927 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 19008
I20260620 18:33:05.205319 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:35671
--webserver_interface=127.10.131.62
--webserver_port=35237
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:36025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:35671
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:33:05.527484 19513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:33:05.527750 19513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:33:05.527937 19513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:33:05.535061 19513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:33:05.535181 19513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:33:05.535257 19513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:33:05.535339 19513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:33:05.535399 19513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:33:05.544975 19513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:36025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:35671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:35671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=35237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:33:05.546274 19513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:33:05.547577 19513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:33:05.559605 19521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:05.559720 19518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:05.559659 19519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:33:05.560577 19513 server_base.cc:1034] running on GCE node
I20260620 18:33:05.561416 19513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:33:05.562842 19513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:33:05.564043 19513 hybrid_clock.cc:648] HybridClock initialized: now 1781980385564000 us; error 39 us; skew 500 ppm
I20260620 18:33:05.564429 19513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:33:05 dist-test-slave-cbn3 krb5kdc[18995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980385, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:33:05.567643 19513 init.cc:377] Logged in from keytab as kudu/127.10.131.62@KRBTEST.COM (short username kudu)
I20260620 18:33:05.569337 19513 webserver.cc:460] Webserver started at http://127.10.131.62:35237/ using document root <none> and password file <none>
I20260620 18:33:05.570046 19513 fs_manager.cc:359] Metadata directory not provided
I20260620 18:33:05.570207 19513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:33:05.575709 19513 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 18:33:05.578527 19528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:05.579828 19513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:33:05.579991 19513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "c7407a912a9b44a48ac5ad1fe746d21f"
format_stamp: "Formatted at 2026-06-20 18:33:02 on dist-test-slave-cbn3"
server_key: "21f543ff36d741b9da84346ecfab4756"
server_key_iv: "87b107037ee8372261b4269f7fe187e6"
server_key_version: "encryptionkey@0"
I20260620 18:33:05.581017 19513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:33:05.605712 19513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:33:05.610631 19513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:33:05.610961 19513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:33:05.637153 19513 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:35671
I20260620 18:33:05.637172 19579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:35671 every 8 connection(s)
I20260620 18:33:05.638617 19513 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:33:05.645391 19580 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:33:05.647619 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 19513
I20260620 18:33:05.648608 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.1:37579
--local_ip_for_outbound_sockets=127.10.131.1
--tserver_master_addrs=127.10.131.62:35671
--webserver_port=33937
--webserver_interface=127.10.131.1
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:36025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.131.1
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:33:05.649075 19580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f: Bootstrap starting.
I20260620 18:33:05.669459 19580 log.cc:824] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:33:05.678397 19580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:33:05.678918 19580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f: Bootstrap complete.
I20260620 18:33:05.688355 19580 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7407a912a9b44a48ac5ad1fe746d21f" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35671 } }
I20260620 18:33:05.689273 19580 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7407a912a9b44a48ac5ad1fe746d21f, State: Initialized, Role: FOLLOWER
I20260620 18:33:05.689957 19580 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7407a912a9b44a48ac5ad1fe746d21f" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35671 } }
I20260620 18:33:05.690201 19580 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:33:05.690310 19580 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:33:05.690498 19580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:33:05.694113 19580 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7407a912a9b44a48ac5ad1fe746d21f" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35671 } }
I20260620 18:33:05.694581 19580 leader_election.cc:302] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7407a912a9b44a48ac5ad1fe746d21f; no voters:
I20260620 18:33:05.695240 19580 leader_election.cc:288] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 18:33:05.695369 19584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f [term 2 FOLLOWER]: Leader election won for term 2
I20260620 18:33:05.696578 19584 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f [term 2 LEADER]: Becoming Leader. State: Replica: c7407a912a9b44a48ac5ad1fe746d21f, State: Running, Role: LEADER
I20260620 18:33:05.697202 19584 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7407a912a9b44a48ac5ad1fe746d21f" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35671 } }
I20260620 18:33:05.697470 19580 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:33:05.700213 19585 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c7407a912a9b44a48ac5ad1fe746d21f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7407a912a9b44a48ac5ad1fe746d21f" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35671 } } }
I20260620 18:33:05.700702 19585 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f [sys.catalog]: This master's current role is: LEADER
I20260620 18:33:05.701087 19586 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7407a912a9b44a48ac5ad1fe746d21f. Latest consensus state: current_term: 2 leader_uuid: "c7407a912a9b44a48ac5ad1fe746d21f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7407a912a9b44a48ac5ad1fe746d21f" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35671 } } }
I20260620 18:33:05.701447 19586 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f [sys.catalog]: This master's current role is: LEADER
I20260620 18:33:05.709014 19590 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:33:05.714716 19590 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:33:05.716094 19590 catalog_manager.cc:1259] Loaded cluster ID: d49bfd83e27c40c3a874c1ff34cb7565
I20260620 18:33:05.716259 19590 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:33:05.720038 19590 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:33:05.721517 19590 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c7407a912a9b44a48ac5ad1fe746d21f: Loaded TSK: 0
I20260620 18:33:05.723066 19590 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 18:33:05.985991 19582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:33:05.986285 19582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:33:05.986450 19582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:33:05.993716 19582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:33:05.993863 19582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:33:05.994032 19582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.1
I20260620 18:33:06.003798 19582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:36025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.1.keytab
--principal=oryx/127.10.131.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.1:37579
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.131.1
--webserver_port=33937
--tserver_master_addrs=127.10.131.62:35671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.1
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:33:06.005137 19582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:33:06.006399 19582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:33:06.020613 19606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:06.020586 19607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:06.021240 19609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:33:06.021534 19582 server_base.cc:1034] running on GCE node
I20260620 18:33:06.022285 19582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:33:06.023429 19582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:33:06.024693 19582 hybrid_clock.cc:648] HybridClock initialized: now 1781980386024622 us; error 62 us; skew 500 ppm
I20260620 18:33:06.025070 19582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:33:06 dist-test-slave-cbn3 krb5kdc[18995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980386, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:33:06.028673 19582 init.cc:377] Logged in from keytab as oryx/127.10.131.1@KRBTEST.COM (short username oryx)
I20260620 18:33:06.030300 19582 webserver.cc:460] Webserver started at http://127.10.131.1:33937/ using document root <none> and password file <none>
I20260620 18:33:06.030994 19582 fs_manager.cc:359] Metadata directory not provided
I20260620 18:33:06.031154 19582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:33:06.036679 19582 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 18:33:06.039525 19616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:06.040975 19582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:33:06.041144 19582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-0/wal
uuid: "13c284cb03f44a0184b459a0d8feb06d"
format_stamp: "Formatted at 2026-06-20 18:33:03 on dist-test-slave-cbn3"
server_key: "f7730063a6be2db8f94ab44c0347d172"
server_key_iv: "14849ef3d41bf661f3c73571a73cc411"
server_key_version: "encryptionkey@0"
I20260620 18:33:06.042100 19582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:33:06.059496 19582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:33:06.063783 19582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:33:06.064090 19582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:33:06.065119 19582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:33:06.066428 19582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:33:06.066556 19582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:06.066722 19582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:33:06.066821 19582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:06.099150 19582 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.1:37579
I20260620 18:33:06.099246 19728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.1:37579 every 8 connection(s)
I20260620 18:33:06.100580 19582 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
I20260620 18:33:06.102627 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 19582
I20260620 18:33:06.103350 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.2:35765
--local_ip_for_outbound_sockets=127.10.131.2
--tserver_master_addrs=127.10.131.62:35671
--webserver_port=34359
--webserver_interface=127.10.131.2
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:36025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.131.2
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:33:06 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:06 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 18:33:06.517374 19733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:33:06.517644 19733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:33:06.517792 19733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:33:06.524973 19733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:33:06.525066 19733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:33:06.525292 19733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.2
I20260620 18:33:06.535238 19733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:36025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.2.keytab
--principal=oryx/127.10.131.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.2:35765
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.131.2
--webserver_port=34359
--tserver_master_addrs=127.10.131.62:35671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.2
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:33:06.536618 19733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:33:06.538005 19733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:33:06.552063 19739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:06.552142 19740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:06.552631 19742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:33:06.554219 19733 server_base.cc:1034] running on GCE node
I20260620 18:33:06.555305 19733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:33:06.557150 19733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:33:06.558461 19733 hybrid_clock.cc:648] HybridClock initialized: now 1781980386558391 us; error 52 us; skew 500 ppm
I20260620 18:33:06.559017 19733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:33:06 dist-test-slave-cbn3 krb5kdc[18995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980386, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:33:06.564435 19733 init.cc:377] Logged in from keytab as oryx/127.10.131.2@KRBTEST.COM (short username oryx)
I20260620 18:33:06.566820 19733 webserver.cc:460] Webserver started at http://127.10.131.2:34359/ using document root <none> and password file <none>
I20260620 18:33:06.568166 19733 fs_manager.cc:359] Metadata directory not provided
I20260620 18:33:06.568374 19733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:33:06.578471 19733 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.008s sys 0.000s
I20260620 18:33:06.582937 19749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:06.585253 19733 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 18:33:06.585507 19733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-1/wal
uuid: "bccbcc551de749f0955340b0a734d512"
format_stamp: "Formatted at 2026-06-20 18:33:03 on dist-test-slave-cbn3"
server_key: "28bc36fa85f584a946504d5b0f565590"
server_key_iv: "c034ba5ac770e048096673c5706cc59f"
server_key_version: "encryptionkey@0"
I20260620 18:33:06.587425 19733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:33:06.657183 19733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:33:06.664642 19733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:33:06.665099 19733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:33:06.666765 19733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:33:06.669190 19733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:33:06.669348 19733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:06.669502 19733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:33:06.669610 19733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:06.711035 19733 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.2:35765
I20260620 18:33:06.711071 19861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.2:35765 every 8 connection(s)
I20260620 18:33:06.713568 19733 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
I20260620 18:33:06.719861 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 19733
I20260620 18:33:06.720585 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.3:40125
--local_ip_for_outbound_sockets=127.10.131.3
--tserver_master_addrs=127.10.131.62:35671
--webserver_port=34805
--webserver_interface=127.10.131.3
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:36025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.131.3
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:33:06 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.2@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:06 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.2@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 18:33:07.054711 19866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:33:07.055027 19866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:33:07.055202 19866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:33:07.062505 19866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:33:07.062652 19866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:33:07.062845 19866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.3
I20260620 18:33:07.072523 19866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:36025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/krb5kdc/oryx_127.10.131.3.keytab
--principal=oryx/127.10.131.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.3:40125
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.131.3
--webserver_port=34805
--tserver_master_addrs=127.10.131.62:35671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.3
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:33:07.073877 19866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:33:07.075181 19866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:33:07.093571 19872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:07.093874 19873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:07.099601 19876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:33:07.099849 19866 server_base.cc:1034] running on GCE node
I20260620 18:33:07.100746 19866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:33:07.110728 19866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:33:07.112044 19866 hybrid_clock.cc:648] HybridClock initialized: now 1781980387111948 us; error 93 us; skew 500 ppm
I20260620 18:33:07.112399 19866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:33:07 dist-test-slave-cbn3 krb5kdc[18995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980387, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.131.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:33:07.115793 19866 init.cc:377] Logged in from keytab as oryx/127.10.131.3@KRBTEST.COM (short username oryx)
I20260620 18:33:07.118350 19866 webserver.cc:460] Webserver started at http://127.10.131.3:34805/ using document root <none> and password file <none>
I20260620 18:33:07.119057 19866 fs_manager.cc:359] Metadata directory not provided
I20260620 18:33:07.119217 19866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:33:07.124732 19866 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 18:33:07.127525 19882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:07.129015 19866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:33:07.129189 19866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-2/wal
uuid: "cc741f493db34965aa402be52638c2f5"
format_stamp: "Formatted at 2026-06-20 18:33:04 on dist-test-slave-cbn3"
server_key: "b82ef8e5028bcdaec3c28934a4940608"
server_key_iv: "0d247c1ab9b91482032d8002b8f107fa"
server_key_version: "encryptionkey@0"
I20260620 18:33:07.130105 19866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:33:07.152907 19866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:33:07.157434 19866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:33:07.157701 19866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:33:07.158767 19866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:33:07.160243 19866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:33:07.160369 19866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:07.160507 19866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:33:07.160606 19866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:07.205708 19866 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.3:40125
I20260620 18:33:07.205755 19994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.3:40125 every 8 connection(s)
I20260620 18:33:07.207221 19866 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
I20260620 18:33:07.216480 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 19866
Jun 20 18:33:07 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.3@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:07 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.3@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:07.269913 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:07.217413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54656 (local address 127.10.131.62:35671)
0620 18:33:07.226993 (+ 9580us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:07.227003 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:07.236454 (+ 9451us) server_negotiation.cc:407] Connection header received
0620 18:33:07.236699 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:07.236708 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:07.236835 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:07.237071 (+ 236us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:07.241191 (+ 4120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:07.242893 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:07.247377 (+ 4484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:07.248368 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:07.248547 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:07.253651 (+ 5104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:07.253667 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:07.253704 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:33:07.253783 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:07.261549 (+ 7766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:07.262941 (+ 1392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:07.262956 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:07.262972 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:07.263286 (+ 314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:07.263907 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:07.263915 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:07.263920 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:07.264866 (+ 946us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:07.265322 (+ 456us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:07.267462 (+ 2140us) server_negotiation.cc:299] Negotiation successful
0620 18:33:07.268154 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9402,"thread_start_us":9233,"threads_started":1}
W20260620 18:33:07.448057 19732 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.332s user 0.004s sys 0.009s
W20260620 18:33:07.449424 19731 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:06.103774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:35207 (local address 127.10.131.62:35671)
0620 18:33:06.104451 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:06.104462 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:06.105187 (+ 725us) server_negotiation.cc:407] Connection header received
0620 18:33:06.107332 (+ 2145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:06.107344 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:06.107657 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:06.108035 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:06.109858 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:06.111209 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:06.113831 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:06.114474 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:06.114603 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:07.449124 (+1334521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.1:35207: BlockingRecv error: recv got EOF from 127.10.131.1:35207 (error 108)
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":261,"threads_started":1}
I20260620 18:33:07.450042 19732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:06.103851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.1:35207)
0620 18:33:06.105004 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:06.105051 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:06.106780 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:06.108180 (+ 1400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:06.108192 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:06.108805 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:06.109651 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:06.109679 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:06.111405 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:06.111414 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:06.113683 (+ 2269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:06.113703 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:06.114727 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:06.114738 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:06.115106 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:06.115875 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:06.115912 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:07.448850 (+1332938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":835,"thread_start_us":650,"threads_started":1}
W20260620 18:33:07.450814 19729 heartbeater.cc:644] Failed to heartbeat to 127.10.131.62:35671 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.131.62:35671: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Jun 20 18:33:07 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:07 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:07.460965 19732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:07.451484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.1:52835)
0620 18:33:07.451749 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:07.451769 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:07.451891 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:07.452237 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:07.452242 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:07.452634 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:07.452795 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:07.452804 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:07.453826 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:07.453831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:07.454838 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:07.454850 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:07.455554 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:07.455560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:07.455727 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:07.456341 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:07.456359 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:07.460640 (+ 4281us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141}
W20260620 18:33:07.461072 19731 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:07.451644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:52835 (local address 127.10.131.62:35671)
0620 18:33:07.451788 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:07.451795 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:07.451814 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:33:07.452021 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:07.452026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:07.452108 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:07.452207 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:07.452953 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:07.453681 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:07.454999 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:07.455384 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:07.455528 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:07.460815 (+ 5287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.1:52835: BlockingRecv error: recv got EOF from 127.10.131.1:52835 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 18:33:07 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:07 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 18:33:07.470580 19731 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:07.461851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:56161 (local address 127.10.131.62:35671)
0620 18:33:07.461996 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:07.462000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:07.462170 (+ 170us) server_negotiation.cc:407] Connection header received
0620 18:33:07.462344 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:07.462350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:07.462417 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:07.462517 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:07.463135 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:07.463843 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:07.465215 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:07.465654 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:07.465787 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:07.470325 (+ 4538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.1:56161: BlockingRecv error: recv got EOF from 127.10.131.1:56161 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 18:33:07.471596 19732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:07.461795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.1:56161)
0620 18:33:07.462086 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:07.462100 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:07.462228 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:07.462550 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:07.462557 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:07.462848 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:07.462998 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:07.463006 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:07.463987 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:07.463993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:07.465073 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:07.465084 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:07.465809 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:07.465815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:07.466001 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:07.466615 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:07.466632 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:07.471242 (+ 4610us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217}
W20260620 18:33:07.471880 19729 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:33:08.139370 19865 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.410s user 0.005s sys 0.013s
W20260620 18:33:08.141018 19864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:06.716499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:57151 (local address 127.10.131.62:35671)
0620 18:33:06.717043 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:06.717049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:06.718010 (+ 961us) server_negotiation.cc:407] Connection header received
0620 18:33:06.720104 (+ 2094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:06.720114 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:06.720230 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:06.720417 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:06.722584 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:06.724306 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:06.727034 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:06.727999 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:06.728181 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:08.140730 (+1412549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.2:57151: BlockingRecv error: recv got EOF from 127.10.131.2:57151 (error 108)
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":258,"threads_started":1}
I20260620 18:33:08.142293 19865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:06.717003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.2:57151)
0620 18:33:06.717819 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:06.717878 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:06.719714 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:06.720695 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:06.720708 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:06.721520 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:06.722346 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:06.722371 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:06.724524 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:06.724533 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:06.726825 (+ 2292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:06.726854 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:06.728271 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:06.728280 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:06.728626 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:06.729611 (+ 985us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:06.729645 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:08.140629 (+1410984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":333,"threads_started":1}
W20260620 18:33:08.143360 19862 heartbeater.cc:644] Failed to heartbeat to 127.10.131.62:35671 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.131.62:35671: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Jun 20 18:33:08 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.2@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:08 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.2@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:08.159731 19865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:08.144152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.2:49567)
0620 18:33:08.144675 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:08.144704 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:08.144897 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:08.145297 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:08.145305 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:08.145998 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:08.146287 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:08.146302 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:08.147403 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:08.147411 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:08.149752 (+ 2341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:08.149773 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:08.150583 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:08.150592 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:08.150870 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:08.154819 (+ 3949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:08.154856 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:08.159523 (+ 4667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":244}
W20260620 18:33:08.160213 19864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:08.144204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:49567 (local address 127.10.131.62:35671)
0620 18:33:08.144382 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:08.144405 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:08.144799 (+ 394us) server_negotiation.cc:407] Connection header received
0620 18:33:08.145057 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:08.145062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:08.145135 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:08.145217 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:08.146451 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:08.147255 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:08.149939 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:08.150368 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:08.150467 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:08.159878 (+ 9411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.2:49567: BlockingRecv error: recv got EOF from 127.10.131.2:49567 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 18:33:08 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.2@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:08 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.2@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:08.171619 20002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:08.160420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.2:53559)
0620 18:33:08.161009 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:08.161029 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:08.161145 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:08.161542 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:08.161545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:08.161947 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:08.162161 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:08.162168 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:08.164050 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:08.164054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:08.165092 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:08.165103 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:08.166444 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:08.166451 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:08.166640 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:08.167378 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:08.167425 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:08.171407 (+ 3982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":384,"threads_started":1}
W20260620 18:33:08.171943 19862 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:33:08.172161 19864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:08.160758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:53559 (local address 127.10.131.62:35671)
0620 18:33:08.161004 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:08.161013 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:08.161093 (+ 80us) server_negotiation.cc:407] Connection header received
0620 18:33:08.161265 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:08.161273 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:08.161379 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:08.161520 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:08.162300 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:08.163869 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:08.165230 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:08.166253 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:08.166418 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:08.171785 (+ 5367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.2:53559: BlockingRecv error: recv got EOF from 127.10.131.2:53559 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 18:33:08.466308 19997 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.223s user 0.006s sys 0.019s
W20260620 18:33:08.467386 19998 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:07.211873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:36591 (local address 127.10.131.62:35671)
0620 18:33:07.217412 (+ 5539us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:07.217422 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:07.221928 (+ 4506us) server_negotiation.cc:407] Connection header received
0620 18:33:07.225936 (+ 4008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:07.225942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:07.226051 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:07.226209 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:07.230174 (+ 3965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:07.232034 (+ 1860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:07.236742 (+ 4708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:07.237847 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:07.237997 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:08.467170 (+1229173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.3:36591: BlockingRecv error: recv got EOF from 127.10.131.3:36591 (error 108)
Metrics: {"server-negotiator.queue_time_us":5310,"thread_start_us":5170,"threads_started":1}
I20260620 18:33:08.467897 19997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:07.211133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.3:36591)
0620 18:33:07.221674 (+ 10541us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:07.221755 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:07.225511 (+ 3756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:07.227539 (+ 2028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:07.227557 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:07.228644 (+ 1087us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:07.229869 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:07.229905 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:07.232364 (+ 2459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:07.232374 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:07.235457 (+ 3083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:07.235481 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:07.241468 (+ 5987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:07.241478 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:07.241970 (+ 492us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:07.243036 (+ 1066us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:07.243102 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:08.466972 (+1223870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":10063,"thread_start_us":9788,"threads_started":1}
W20260620 18:33:08.468937 19995 heartbeater.cc:644] Failed to heartbeat to 127.10.131.62:35671 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.131.62:35671: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Jun 20 18:33:08 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.3@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:08 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.3@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:08 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:08.479350 19997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:08.469572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.3:50411)
0620 18:33:08.469826 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:08.469846 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:08.469970 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:08.470359 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:08.470362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:08.470840 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:08.471057 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:08.471085 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:08.472239 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:08.472245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:08.473512 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:08.473524 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:08.474263 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:08.474269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:08.474432 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:08.475048 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:08.475064 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:08.478994 (+ 3930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96}
W20260620 18:33:08.479471 19998 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:08.469710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:50411 (local address 127.10.131.62:35671)
0620 18:33:08.469877 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:08.469885 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:08.469911 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:33:08.470099 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:08.470103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:08.470178 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:08.470291 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:08.471199 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:08.472071 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:08.473662 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:08.474084 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:08.474282 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:08.479279 (+ 4997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.3:50411: BlockingRecv error: recv got EOF from 127.10.131.3:50411 (error 108)
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":46}
Jun 20 18:33:08 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:08.482018 20003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:08.472488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.1:48007)
0620 18:33:08.473041 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:08.473059 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:08.473205 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:08.473521 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:08.473526 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:08.473916 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:08.474040 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:08.474048 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:08.475162 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:08.475168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:08.476151 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:08.476162 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:08.476955 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:08.476961 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:08.477117 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:08.477639 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:08.477666 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:08.481696 (+ 4030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":342,"threads_started":1}
W20260620 18:33:08.482175 19864 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:08.472534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:48007 (local address 127.10.131.62:35671)
0620 18:33:08.472793 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:08.472801 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:08.473141 (+ 340us) server_negotiation.cc:407] Connection header received
0620 18:33:08.473322 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:08.473327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:08.473399 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:08.473494 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:08.474179 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:08.475030 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:08.476303 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:08.476767 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:08.476920 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:08.481875 (+ 4955us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.1:48007: BlockingRecv error: recv got EOF from 127.10.131.1:48007 (error 108)
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":118}
Jun 20 18:33:08 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.3@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:08 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.3@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:08.488653 19997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:08.480217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.3:35667)
0620 18:33:08.480461 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:08.480480 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:08.480582 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:08.480927 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:08.480932 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:08.481330 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:08.481513 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:08.481525 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:08.482718 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:08.482723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:08.483872 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:08.483884 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:08.484589 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:08.484593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:08.484767 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:08.485285 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:08.485301 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:08.488463 (+ 3162us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109}
W20260620 18:33:08.488971 19995 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:33:08.489228 19998 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:08.480337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:35667 (local address 127.10.131.62:35671)
0620 18:33:08.480467 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:08.480471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:08.480545 (+ 74us) server_negotiation.cc:407] Connection header received
0620 18:33:08.480692 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:08.480695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:08.480768 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:08.480884 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:08.481633 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:08.482586 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:08.483981 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:08.484454 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:08.484564 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:08.488667 (+ 4103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.3:35667: BlockingRecv error: recv got EOF from 127.10.131.3:35667 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 18:33:09 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.2@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:09 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.2@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:09.181546 20004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:09.172504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.2:37923)
0620 18:33:09.173206 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:09.173227 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:09.173416 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:09.173813 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:09.173818 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:09.174237 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:09.174432 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:09.174447 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:09.175750 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:09.175754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:09.176655 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:09.176666 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:09.177388 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:09.177392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:09.177510 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:09.178064 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:09.178081 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:09.181334 (+ 3253us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":445,"threads_started":1}
W20260620 18:33:09.181780 20005 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:09.172644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:37923 (local address 127.10.131.62:35671)
0620 18:33:09.173152 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:09.173156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:09.173351 (+ 195us) server_negotiation.cc:407] Connection header received
0620 18:33:09.173533 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:09.173538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:09.173615 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:09.173739 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:09.174608 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:09.175609 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:09.176772 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:09.177262 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:09.177352 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:09.181511 (+ 4159us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.2:37923: BlockingRecv error: recv got EOF from 127.10.131.2:37923 (error 108)
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":287,"threads_started":1}
Jun 20 18:33:09 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:09 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:09.493291 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:09.483617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.1:34261)
0620 18:33:09.484120 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:09.484151 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:09.484265 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:09.484545 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:09.484548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:09.485035 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:09.485249 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:09.485263 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:09.486245 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:09.486249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:09.487168 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:09.487179 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:09.487830 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:09.487834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:09.487949 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:09.488638 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:09.488665 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:09.492947 (+ 4282us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":304,"threads_started":1}
W20260620 18:33:09.493372 20005 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:09.483660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:34261 (local address 127.10.131.62:35671)
0620 18:33:09.483880 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:09.483889 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:09.484230 (+ 341us) server_negotiation.cc:407] Connection header received
0620 18:33:09.484360 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:09.484364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:09.484423 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:09.484498 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:09.485383 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:09.486113 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:09.487299 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:09.487706 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:09.487796 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:09.493112 (+ 5316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.1:34261: BlockingRecv error: recv got EOF from 127.10.131.1:34261 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 18:33:09 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.3@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:09 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.3@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:09.499452 20007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:09.489525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.3:48603)
0620 18:33:09.490760 (+ 1235us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:09.490779 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:09.490892 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:09.491220 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:09.491227 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:09.491636 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:09.491791 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:09.491800 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:09.493250 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:09.493255 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:09.494309 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:09.494322 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:09.495035 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:09.495039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:09.495153 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:09.495777 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:09.495795 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:09.499228 (+ 3433us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1120,"thread_start_us":841,"threads_started":1}
W20260620 18:33:09.499676 20008 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:09.489604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:48603 (local address 127.10.131.62:35671)
0620 18:33:09.490928 (+ 1324us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:09.490931 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:09.490945 (+ 14us) server_negotiation.cc:407] Connection header received
0620 18:33:09.490980 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:09.490984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:09.491044 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:09.491122 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:09.491936 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:09.493103 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:09.494414 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:09.494888 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:09.494977 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:09.499426 (+ 4449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.3:48603: BlockingRecv error: recv got EOF from 127.10.131.3:48603 (error 108)
Metrics: {"server-negotiator.queue_time_us":1220,"thread_start_us":1088,"threads_started":1}
Jun 20 18:33:10 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.2@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:10 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.2@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:10.192466 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:10.182626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.2:50739)
0620 18:33:10.183178 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:10.183210 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:10.183324 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:10.183592 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:10.183595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:10.184062 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:10.184233 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:10.184244 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:10.185556 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:10.185560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:10.186617 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:10.186628 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:10.187321 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:10.187325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:10.187458 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:10.188021 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:10.188039 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:10.192067 (+ 4028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":315,"threads_started":1}
W20260620 18:33:10.192574 20010 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:10.182678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:50739 (local address 127.10.131.62:35671)
0620 18:33:10.183174 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:10.183179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:10.183263 (+ 84us) server_negotiation.cc:407] Connection header received
0620 18:33:10.183406 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:10.183409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:10.183466 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:10.183549 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:10.184409 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:10.185413 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:10.186754 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:10.187186 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:10.187302 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:10.192282 (+ 4980us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.2:50739: BlockingRecv error: recv got EOF from 127.10.131.2:50739 (error 108)
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":285,"threads_started":1}
Jun 20 18:33:10 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:10 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:10.504046 20011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:10.494344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.1:54177)
0620 18:33:10.494919 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:10.494941 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:10.495110 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:10.495477 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:10.495482 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:10.495913 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:10.496091 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:10.496100 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:10.497241 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:10.497245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:10.498279 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:10.498294 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:10.498967 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:10.498972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:10.499119 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:10.499717 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:10.499737 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:10.503702 (+ 3965us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":298,"threads_started":1}
W20260620 18:33:10.504133 20010 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:10.494585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:54177 (local address 127.10.131.62:35671)
0620 18:33:10.494772 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:10.494780 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:10.495025 (+ 245us) server_negotiation.cc:407] Connection header received
0620 18:33:10.495248 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:10.495253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:10.495333 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:10.495443 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:10.496227 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:10.497070 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:10.498426 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:10.498833 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:10.498932 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:10.503905 (+ 4973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.1:54177: BlockingRecv error: recv got EOF from 127.10.131.1:54177 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 18:33:10 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.3@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:10 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.3@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:10.510329 20012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:10.500429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.3:42867)
0620 18:33:10.500939 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:10.500960 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:10.501095 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:10.501531 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:10.501537 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:10.501978 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:10.502147 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:10.502155 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:10.503186 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:10.503192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:10.504090 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:10.504101 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:10.505123 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:10.505130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:10.505314 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:10.505833 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:10.505849 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:10.509998 (+ 4149us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":266,"threads_started":1}
W20260620 18:33:10.510537 20013 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:10.500498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:42867 (local address 127.10.131.62:35671)
0620 18:33:10.501248 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:10.501252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:10.501264 (+ 12us) server_negotiation.cc:407] Connection header received
0620 18:33:10.501299 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:10.501302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:10.501351 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:10.501491 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:10.502283 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:10.503059 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:10.504498 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:10.504976 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:10.505107 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:10.510250 (+ 5143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.3:42867: BlockingRecv error: recv got EOF from 127.10.131.3:42867 (error 108)
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":489,"threads_started":1}
Jun 20 18:33:11 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.2@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:11 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.2@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:11.203071 20014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:11.193454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.2:55875)
0620 18:33:11.194249 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:11.194271 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:11.194401 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:11.194743 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:11.194747 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:11.195190 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:11.195391 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:11.195402 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:11.196529 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:11.196533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:11.197655 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:11.197670 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:11.198362 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:11.198367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:11.198505 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:11.199246 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:11.199265 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:11.202712 (+ 3447us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":318,"threads_started":1}
W20260620 18:33:11.203197 20015 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:11.193557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:55875 (local address 127.10.131.62:35671)
0620 18:33:11.194369 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:11.194374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:11.194391 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:33:11.194507 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:11.194511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:11.194569 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:11.194706 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:11.195527 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:11.196368 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:11.197783 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:11.198238 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:11.198331 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:11.202916 (+ 4585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.2:55875: BlockingRecv error: recv got EOF from 127.10.131.2:55875 (error 108)
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":581,"threads_started":1}
Jun 20 18:33:11 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:11 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:11.515174 20016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:11.505195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.1:38955)
0620 18:33:11.505762 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:11.505782 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:11.505885 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:11.506167 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:11.506170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:11.506523 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:11.506709 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:11.506718 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:11.507692 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:11.507696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:11.508706 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:11.508717 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:11.509499 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:11.509504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:11.509635 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:11.510357 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:11.510375 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:11.514800 (+ 4425us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":288,"threads_started":1}
W20260620 18:33:11.515232 20015 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:11.505245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:38955 (local address 127.10.131.62:35671)
0620 18:33:11.505400 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:11.505404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:11.505847 (+ 443us) server_negotiation.cc:407] Connection header received
0620 18:33:11.505975 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:11.505979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:11.506042 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:11.506122 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:11.506829 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:11.507553 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:11.508830 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:11.509306 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:11.509400 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:11.514985 (+ 5585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.1:38955: BlockingRecv error: recv got EOF from 127.10.131.1:38955 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 18:33:11 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.3@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:11 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.3@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:11.522619 20017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:11.511379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.3:48319)
0620 18:33:11.512030 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:11.512052 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:11.512182 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:11.512588 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:11.512618 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:11.513134 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:11.513389 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:11.513405 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:11.515917 (+ 2512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:11.515924 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:11.517293 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:11.517310 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:11.517982 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:11.517988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:11.518190 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:11.518734 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:11.518751 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:11.522239 (+ 3488us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":278,"threads_started":1}
W20260620 18:33:11.522761 20018 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:11.511472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:48319 (local address 127.10.131.62:35671)
0620 18:33:11.512254 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:11.512259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:11.512278 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:33:11.512324 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:11.512331 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:11.512400 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:11.512509 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:11.513558 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:11.514679 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:11.517421 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:11.517832 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:11.517964 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:11.522473 (+ 4509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.3:48319: BlockingRecv error: recv got EOF from 127.10.131.3:48319 (error 108)
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":562,"threads_started":1}
Jun 20 18:33:12 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.2@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:12 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.2@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:12.213748 20019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:12.204013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.2:54123)
0620 18:33:12.204489 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:12.204539 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:12.204647 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:12.205158 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:12.205169 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:12.205549 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:12.205758 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:12.205769 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:12.207201 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:12.207205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:12.208223 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:12.208236 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:12.208980 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:12.208984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:12.209111 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:12.209784 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:12.209802 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:12.213364 (+ 3562us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":265,"threads_started":1}
W20260620 18:33:12.213874 20020 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:12.204120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:54123 (local address 127.10.131.62:35671)
0620 18:33:12.204888 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:12.204893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:12.204908 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:33:12.204954 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:12.204957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:12.205015 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:12.205100 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:12.205909 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:12.207034 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:12.208365 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:12.208790 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:12.208961 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:12.213621 (+ 4660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.2:54123: BlockingRecv error: recv got EOF from 127.10.131.2:54123 (error 108)
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":533,"threads_started":1}
Jun 20 18:33:12 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:12 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 18:33:12.533841 20020 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:12.518106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:53373 (local address 127.10.131.62:35671)
0620 18:33:12.518299 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:12.518305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:12.518736 (+ 431us) server_negotiation.cc:407] Connection header received
0620 18:33:12.519113 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:12.519121 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:12.519213 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:12.519335 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:12.520523 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:12.521911 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:12.523403 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:12.524079 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:12.524219 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:12.533604 (+ 9385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.1:53373: BlockingRecv error: recv got EOF from 127.10.131.1:53373 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 18:33:12.534739 20021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:12.517948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.1:53373)
0620 18:33:12.518595 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:12.518626 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:12.518783 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:12.519387 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:12.519392 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:12.520059 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:12.520306 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:12.520320 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:12.522098 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:12.522102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:12.523195 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:12.523208 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:12.524823 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:12.524830 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:12.525093 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:12.525993 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:12.526014 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:12.534420 (+ 8406us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":344,"threads_started":1}
Jun 20 18:33:12 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.3@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:12 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.3@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:12.538447 20022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:12.524174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.3:43609)
0620 18:33:12.524939 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:12.524971 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:12.525135 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:12.525619 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:12.525627 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:12.526165 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:12.526366 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:12.526378 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:12.528200 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:12.528207 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:12.530452 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:12.530473 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:12.531496 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:12.531502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:12.531730 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:12.533561 (+ 1831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:12.533590 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:12.538127 (+ 4537us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":323,"threads_started":1}
W20260620 18:33:12.538594 20023 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:12.524307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:43609 (local address 127.10.131.62:35671)
0620 18:33:12.524992 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:12.525000 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:12.525025 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:33:12.525284 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:12.525292 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:12.525416 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:12.525547 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:12.526554 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:12.527971 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:12.530656 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:12.531286 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:12.531417 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:12.538347 (+ 6930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.3:43609: BlockingRecv error: recv got EOF from 127.10.131.3:43609 (error 108)
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":313,"threads_started":1}
Jun 20 18:33:13 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.2@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:13 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.2@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:13.229542 20024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:13.214940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.2:33879)
0620 18:33:13.215486 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:13.215505 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:13.215623 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:13.215977 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:13.215980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:13.216416 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:13.216589 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:13.216604 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:13.218029 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:13.218035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:13.219192 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:13.219206 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:13.219869 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:13.219873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:13.220103 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:13.225354 (+ 5251us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:13.225375 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:13.229191 (+ 3816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":290,"threads_started":1}
W20260620 18:33:13.229843 20025 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:13.215165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:33879 (local address 127.10.131.62:35671)
0620 18:33:13.215673 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:13.215677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:13.215696 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:33:13.215752 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:13.215756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:13.215820 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:13.215929 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:13.216781 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:13.217825 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:13.219320 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:13.219730 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:13.219826 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:13.229531 (+ 9705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.2:33879: BlockingRecv error: recv got EOF from 127.10.131.2:33879 (error 108)
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":286,"threads_started":1}
Jun 20 18:33:13 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:13 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:13.546553 20026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:13.535800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.1:35317)
0620 18:33:13.536296 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:13.536316 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:13.536429 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:13.536703 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:13.536706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:13.537180 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:13.537348 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:13.537358 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:13.538534 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:13.538538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:13.539658 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:13.539677 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:13.541273 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:13.541282 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:13.541505 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:13.542163 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:13.542183 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:13.546088 (+ 3905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":283,"threads_started":1}
W20260620 18:33:13.546557 20025 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:13.535895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:35317 (local address 127.10.131.62:35671)
0620 18:33:13.536060 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:13.536064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:13.536388 (+ 324us) server_negotiation.cc:407] Connection header received
0620 18:33:13.536519 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:13.536523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:13.536588 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:13.536666 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:13.537543 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:13.538385 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:13.539807 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:13.540895 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:13.541074 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:13.546316 (+ 5242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.1:35317: BlockingRecv error: recv got EOF from 127.10.131.1:35317 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 18:33:13 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.3@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:13 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.3@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:13.551244 20027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:13.539555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.3:45537)
0620 18:33:13.541236 (+ 1681us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:13.541257 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:13.541365 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:13.542406 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:13.542411 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:13.542840 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:13.542975 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:13.542983 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:13.544024 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:13.544030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:13.545065 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:13.545078 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:13.546413 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:13.546419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:13.546795 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:13.547632 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:13.547650 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:13.550894 (+ 3244us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1554,"thread_start_us":1448,"threads_started":1}
W20260620 18:33:13.551528 20028 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:13.539677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:45537 (local address 127.10.131.62:35671)
0620 18:33:13.542144 (+ 2467us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:13.542148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:13.542162 (+ 14us) server_negotiation.cc:407] Connection header received
0620 18:33:13.542197 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:13.542201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:13.542255 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:13.542384 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:13.543114 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:13.543890 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:13.545209 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:13.545935 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:13.546051 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:13.551128 (+ 5077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.3:45537: BlockingRecv error: recv got EOF from 127.10.131.3:45537 (error 108)
Metrics: {"server-negotiator.queue_time_us":2365,"thread_start_us":2173,"threads_started":1}
Jun 20 18:33:14 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.2@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:14 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.2@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:14.240021 20029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:14.230593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.2:51321)
0620 18:33:14.231343 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:14.231388 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:14.231524 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:14.232120 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:14.232128 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:14.232527 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:14.232770 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:14.232780 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:14.233925 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:14.233929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:14.234956 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:14.234968 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:14.235618 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:14.235622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:14.235737 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:14.236390 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:14.236409 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:14.239784 (+ 3375us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":445,"threads_started":1}
W20260620 18:33:14.240289 20030 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:14.230648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:51321 (local address 127.10.131.62:35671)
0620 18:33:14.231634 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:14.231640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:14.231658 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:33:14.231719 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:14.231724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:14.231795 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:14.231934 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:14.232974 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:14.233794 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:14.235079 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:14.235495 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:14.235591 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:14.240002 (+ 4411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.2:51321: BlockingRecv error: recv got EOF from 127.10.131.2:51321 (error 108)
Metrics: {"server-negotiator.queue_time_us":854,"thread_start_us":747,"threads_started":1}
Jun 20 18:33:14 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:14 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:14.556962 20031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:14.547341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.1:48411)
0620 18:33:14.547933 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:14.547955 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:14.548068 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:14.548368 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:14.548371 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:14.548768 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:14.549025 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:14.549034 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:14.549979 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:14.549982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:14.550930 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:14.550941 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:14.551531 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:14.551535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:14.551714 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:14.552393 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:14.552443 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:14.556613 (+ 4170us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":290,"threads_started":1}
W20260620 18:33:14.557030 20030 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:14.547414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:48411 (local address 127.10.131.62:35671)
0620 18:33:14.547586 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:14.547590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:14.548026 (+ 436us) server_negotiation.cc:407] Connection header received
0620 18:33:14.548172 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:14.548175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:14.548233 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:14.548317 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:14.549145 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:14.549861 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:14.551042 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:14.551411 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:14.551503 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:14.556788 (+ 5285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.1:48411: BlockingRecv error: recv got EOF from 127.10.131.1:48411 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 18:33:14 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.3@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:14 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.3@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:14.562055 20032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:14.552134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.3:49903)
0620 18:33:14.552939 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:14.552957 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:14.553082 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:14.553667 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:14.553674 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:14.554142 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:14.554539 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:14.554571 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:14.555777 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:14.555783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:14.557237 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:14.557272 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:14.557962 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:14.557967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:14.558088 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:14.558605 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:14.558623 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:14.561832 (+ 3209us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":572,"threads_started":1}
W20260620 18:33:14.562323 20033 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:14.552210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:49903 (local address 127.10.131.62:35671)
0620 18:33:14.552779 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:14.552785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:14.553051 (+ 266us) server_negotiation.cc:407] Connection header received
0620 18:33:14.553215 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:14.553219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:14.553304 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:14.553406 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:14.554681 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:14.555622 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:14.557378 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:14.557788 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:14.557877 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:14.562041 (+ 4164us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.3:49903: BlockingRecv error: recv got EOF from 127.10.131.3:49903 (error 108)
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":306,"threads_started":1}
Jun 20 18:33:15 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.2@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:15 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.2@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:15.250257 20034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:15.240933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.2:40741)
0620 18:33:15.241499 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:15.241516 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:15.241639 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:15.242091 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:15.242094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:15.242512 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:15.242703 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:15.242711 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:15.243673 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:15.243676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:15.244584 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:15.244595 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:15.245340 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:15.245344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:15.245496 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:15.246056 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:15.246115 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:15.249898 (+ 3783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":231,"threads_started":1}
W20260620 18:33:15.250349 20035 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:15.241035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:40741 (local address 127.10.131.62:35671)
0620 18:33:15.241816 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:15.241820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:15.241836 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:33:15.241878 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:15.241881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:15.241936 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:15.242067 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:15.242823 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:15.243540 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:15.244708 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:15.245156 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:15.245320 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:15.250084 (+ 4764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.2:40741: BlockingRecv error: recv got EOF from 127.10.131.2:40741 (error 108)
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":580,"threads_started":1}
Jun 20 18:33:15 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:15 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:15.567003 20036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:15.558061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.1:32841)
0620 18:33:15.558618 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:15.558638 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:15.558744 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:15.559092 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:15.559095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:15.559442 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:15.559600 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:15.559608 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:15.560556 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:15.560559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:15.561572 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:15.561583 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:15.562267 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:15.562272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:15.562391 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:15.563079 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:15.563102 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:15.566712 (+ 3610us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":301,"threads_started":1}
W20260620 18:33:15.567206 20035 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:15.558150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:32841 (local address 127.10.131.62:35671)
0620 18:33:15.558319 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:15.558323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:15.558695 (+ 372us) server_negotiation.cc:407] Connection header received
0620 18:33:15.558907 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:15.558914 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:15.558991 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:15.559065 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:15.559713 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:15.560432 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:15.561700 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:15.562088 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:15.562181 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:15.566975 (+ 4794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.1:32841: BlockingRecv error: recv got EOF from 127.10.131.1:32841 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 18:33:15 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.3@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:15 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.3@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:15.572302 20038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:15.562950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.3:47051)
0620 18:33:15.563812 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:15.563838 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:15.563979 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:15.564293 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:15.564298 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:15.564675 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:15.564924 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:15.564934 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:15.565867 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:15.565872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:15.566832 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:15.566849 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:15.567802 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:15.567806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:15.567918 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:15.568473 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:15.568503 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:15.572071 (+ 3568us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":778,"thread_start_us":603,"threads_started":1}
W20260620 18:33:15.572503 20037 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:15.563047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:47051 (local address 127.10.131.62:35671)
0620 18:33:15.563519 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:15.563523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:15.563925 (+ 402us) server_negotiation.cc:407] Connection header received
0620 18:33:15.564085 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:15.564090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:15.564150 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:15.564249 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:15.565049 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:15.565732 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:15.567001 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:15.567669 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:15.567763 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:15.572240 (+ 4477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.3:47051: BlockingRecv error: recv got EOF from 127.10.131.3:47051 (error 108)
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":268,"threads_started":1}
Jun 20 18:33:16 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.2@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:16 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.2@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:16.260398 20039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:16.251233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.2:49439)
0620 18:33:16.251810 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:16.251830 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:16.251936 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:16.252250 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:16.252254 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:16.252615 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:16.252793 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:16.252801 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:16.254011 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:16.254015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:16.254908 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:16.254918 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:16.255582 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:16.255603 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:16.255773 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:16.256541 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:16.256569 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:16.260149 (+ 3580us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":264,"threads_started":1}
W20260620 18:33:16.260705 20040 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:16.251319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:49439 (local address 127.10.131.62:35671)
0620 18:33:16.251854 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:16.251858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:16.251880 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:33:16.252040 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:16.252044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:16.252102 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:16.252188 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:16.252980 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:16.253884 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:16.255021 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:16.255439 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:16.255522 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:16.260375 (+ 4853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.2:49439: BlockingRecv error: recv got EOF from 127.10.131.2:49439 (error 108)
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":308,"threads_started":1}
Jun 20 18:33:16 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:16 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:16.578833 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:16.568279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.1:57271)
0620 18:33:16.568907 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:16.568929 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:16.569040 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:16.569355 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:16.569358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:16.569935 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:16.570084 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:16.570093 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:16.571450 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:16.571455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:16.572497 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:16.572510 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:16.573401 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:16.573410 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:16.573678 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:16.574717 (+ 1039us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:16.574735 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:16.578483 (+ 3748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":350,"threads_started":1}
W20260620 18:33:16.578889 20040 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:16.568397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:57271 (local address 127.10.131.62:35671)
0620 18:33:16.568596 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:16.568601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:16.569004 (+ 403us) server_negotiation.cc:407] Connection header received
0620 18:33:16.569140 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:16.569144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:16.569205 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:16.569295 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:16.570249 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:16.571301 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:16.572651 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:16.573218 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:16.573351 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:16.578667 (+ 5316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.1:57271: BlockingRecv error: recv got EOF from 127.10.131.1:57271 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 18:33:16 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.3@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:16 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.3@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:16.583014 20042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:16.573168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.3:51727)
0620 18:33:16.573973 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:16.573993 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:16.574110 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:16.574577 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:16.574584 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:16.574959 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:16.575111 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:16.575120 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:16.576504 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:16.576509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:16.577533 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:16.577546 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:16.578502 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:16.578509 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:16.578681 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:16.579257 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:16.579277 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:16.582665 (+ 3388us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":471,"threads_started":1}
W20260620 18:33:16.583387 20043 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:16.573240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:51727 (local address 127.10.131.62:35671)
0620 18:33:16.574226 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:16.574233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:16.574276 (+ 43us) server_negotiation.cc:407] Connection header received
0620 18:33:16.574325 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:16.574332 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:16.574416 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:16.574520 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:16.575275 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:16.576376 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:16.577673 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:16.578310 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:16.578452 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:16.582945 (+ 4493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.3:51727: BlockingRecv error: recv got EOF from 127.10.131.3:51727 (error 108)
Metrics: {"server-negotiator.queue_time_us":876,"thread_start_us":715,"threads_started":1}
Jun 20 18:33:17 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.2@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:17 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.2@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:17.270797 20044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:17.261552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.2:53655)
0620 18:33:17.262346 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:17.262363 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:17.262494 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:17.262959 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:17.262966 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:17.263481 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:17.263631 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:17.263639 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:17.264605 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:17.264609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:17.265624 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:17.265635 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:17.266308 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:17.266312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:17.266434 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:17.267007 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:17.267025 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:17.270558 (+ 3533us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":513,"threads_started":1}
W20260620 18:33:17.271049 20045 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:17.261608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:53655 (local address 127.10.131.62:35671)
0620 18:33:17.262499 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:17.262503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:17.262521 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:33:17.262662 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:17.262667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:17.262746 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:17.262907 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:17.263756 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:17.264479 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:17.265746 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:17.266149 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:17.266242 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:17.270761 (+ 4519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.2:53655: BlockingRecv error: recv got EOF from 127.10.131.2:53655 (error 108)
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":426,"threads_started":1}
Jun 20 18:33:17 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:17 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:17.589699 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:17.579925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.1:46985)
0620 18:33:17.580465 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:17.580486 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:17.580599 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:17.580958 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:17.580963 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:17.581310 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:17.581468 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:17.581476 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:17.582438 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:17.582442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:17.583550 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:17.583562 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:17.584657 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:17.584667 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:17.584873 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:17.585583 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:17.585601 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:17.589344 (+ 3743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":308,"threads_started":1}
W20260620 18:33:17.589833 20045 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:17.580063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:46985 (local address 127.10.131.62:35671)
0620 18:33:17.580255 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:17.580259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:17.580588 (+ 329us) server_negotiation.cc:407] Connection header received
0620 18:33:17.580700 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:17.580704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:17.580767 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:17.580958 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:17.581589 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:17.582291 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:17.583708 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:17.584465 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:17.584592 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:17.589580 (+ 4988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.1:46985: BlockingRecv error: recv got EOF from 127.10.131.1:46985 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 20 18:33:17 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.3@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:17 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.3@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:17.594961 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:17.584062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.3:42331)
0620 18:33:17.584981 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:17.584995 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:17.585128 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:17.585494 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:17.585500 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:17.585930 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:17.586124 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:17.586134 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:17.587523 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:17.587527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:17.588553 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:17.588565 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:17.589554 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:17.589560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:17.589736 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:17.590483 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:17.590501 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:17.594629 (+ 4128us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":816,"thread_start_us":634,"threads_started":1}
W20260620 18:33:17.595085 20048 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:17.584064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:42331 (local address 127.10.131.62:35671)
0620 18:33:17.584579 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:17.584585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:17.585073 (+ 488us) server_negotiation.cc:407] Connection header received
0620 18:33:17.585287 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:17.585292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:17.585372 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:17.585466 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:17.586258 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:17.587367 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:17.588712 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:17.589374 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:17.589493 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:17.594827 (+ 5334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.3:42331: BlockingRecv error: recv got EOF from 127.10.131.3:42331 (error 108)
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":237,"threads_started":1}
Jun 20 18:33:18 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.2@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:18 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.2@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:18.284322 20049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:18.271956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.2:54125)
0620 18:33:18.272471 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:18.272495 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:18.272604 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:18.273162 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:18.273166 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:18.273521 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:18.273673 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:18.273682 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:18.274734 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:18.274737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:18.275680 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:18.275692 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:18.276328 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:18.276332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:18.276473 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:18.280228 (+ 3755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:18.280257 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:18.284075 (+ 3818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":253,"threads_started":1}
W20260620 18:33:18.284547 20050 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:18.272040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:54125 (local address 127.10.131.62:35671)
0620 18:33:18.272821 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:18.272826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:18.272884 (+ 58us) server_negotiation.cc:407] Connection header received
0620 18:33:18.272944 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:18.272947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:18.273003 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:18.273090 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:18.273821 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:18.274594 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:18.275806 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:18.276203 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:18.276294 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:18.284300 (+ 8006us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.2:54125: BlockingRecv error: recv got EOF from 127.10.131.2:54125 (error 108)
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":556,"threads_started":1}
Jun 20 18:33:18 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:18 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:18.601374 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:18.590648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.1:51311)
0620 18:33:18.591213 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:18.591234 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:18.591355 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:18.591653 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:18.591656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:18.592041 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:18.592209 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:18.592217 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:18.593395 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:18.593401 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:18.594618 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:18.594631 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:18.595507 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:18.595512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:18.595735 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:18.597211 (+ 1476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:18.597229 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:18.601048 (+ 3819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":339,"threads_started":1}
W20260620 18:33:18.601401 20050 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:18.590854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:51311 (local address 127.10.131.62:35671)
0620 18:33:18.591029 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:18.591034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:18.591352 (+ 318us) server_negotiation.cc:407] Connection header received
0620 18:33:18.591458 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:18.591462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:18.591525 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:18.591614 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:18.592343 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:18.593209 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:18.594763 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:18.595339 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:18.595465 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:18.601238 (+ 5773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.1:51311: BlockingRecv error: recv got EOF from 127.10.131.1:51311 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 18:33:18 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.3@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:18 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.3@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:18.605152 20052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:18.595856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.3:33123)
0620 18:33:18.596454 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:18.596474 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:18.596597 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:18.596881 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:18.596885 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:18.597324 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:18.597580 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:18.597590 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:18.598660 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:18.598668 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:18.600043 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:18.600057 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:18.601037 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:18.601043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:18.601204 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:18.601816 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:18.601833 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:18.604893 (+ 3060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":307,"threads_started":1}
W20260620 18:33:18.605381 20053 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:18.595915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:33123 (local address 127.10.131.62:35671)
0620 18:33:18.596607 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:18.596612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:18.596626 (+ 14us) server_negotiation.cc:407] Connection header received
0620 18:33:18.596661 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:18.596664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:18.596720 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:18.596801 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:18.597725 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:18.598509 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:18.600190 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:18.600883 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:18.601018 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:18.605114 (+ 4096us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.3:33123: BlockingRecv error: recv got EOF from 127.10.131.3:33123 (error 108)
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":484,"threads_started":1}
Jun 20 18:33:19 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.2@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:19 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.2@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:19.295265 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:19.285325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.2:45781)
0620 18:33:19.286053 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:19.286070 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:19.286233 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:19.286566 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:19.286571 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:19.287005 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:19.287177 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:19.287185 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:19.288182 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:19.288190 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:19.289130 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:19.289141 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:19.289842 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:19.289848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:19.290032 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:19.290735 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:19.290754 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:19.295024 (+ 4270us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":511,"threads_started":1}
W20260620 18:33:19.295567 20054 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:19.285375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:45781 (local address 127.10.131.62:35671)
0620 18:33:19.285934 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:19.285939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:19.286162 (+ 223us) server_negotiation.cc:407] Connection header received
0620 18:33:19.286353 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:19.286360 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:19.286426 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:19.286535 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:19.287319 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:19.288044 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:19.289268 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:19.289691 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:19.289825 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:19.295260 (+ 5435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.2:45781: BlockingRecv error: recv got EOF from 127.10.131.2:45781 (error 108)
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":233,"threads_started":1}
Jun 20 18:33:19 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:19 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:19.613759 20056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:19.602330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.1:42525)
0620 18:33:19.602894 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:19.602914 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:19.603025 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:19.603318 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:19.603321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:19.603721 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:19.603883 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:19.603891 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:19.604826 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:19.604829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:19.606412 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:19.606428 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:19.607790 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:19.607800 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:19.608032 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:19.608807 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:19.608837 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:19.613407 (+ 4570us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":342,"threads_started":1}
W20260620 18:33:19.613958 20054 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:19.602436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:42525 (local address 127.10.131.62:35671)
0620 18:33:19.602602 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:19.602606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:19.602986 (+ 380us) server_negotiation.cc:407] Connection header received
0620 18:33:19.603121 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:19.603125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:19.603192 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:19.603274 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:19.604004 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:19.604698 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:19.606556 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:19.607349 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:19.608399 (+ 1050us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:19.613696 (+ 5297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.1:42525: BlockingRecv error: recv got EOF from 127.10.131.1:42525 (error 108)
Metrics: {"mutex_wait_us":886,"server-negotiator.queue_time_us":60}
Jun 20 18:33:19 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.3@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:19 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.3@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:19.618211 20057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:19.606045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.3:41065)
0620 18:33:19.606645 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:19.606660 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:19.606745 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:19.607617 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:19.607623 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:19.608180 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:19.608440 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:19.608454 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:19.610220 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:19.610225 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:19.611771 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:19.611785 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:19.612705 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:19.612711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:19.613078 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:19.613712 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:19.613735 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:19.617880 (+ 4145us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":249,"threads_started":1}
W20260620 18:33:19.618808 20058 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:19.606457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:41065 (local address 127.10.131.62:35671)
0620 18:33:19.607181 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:19.607186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:19.607205 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:33:19.607255 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:19.607259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:19.607331 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:19.607461 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:19.608634 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:19.610046 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:19.611940 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:19.612539 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:19.612667 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:19.618555 (+ 5888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.3:41065: BlockingRecv error: recv got EOF from 127.10.131.3:41065 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":588,"thread_start_us":428,"threads_started":1}
Jun 20 18:33:20 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.2@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:20 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.2@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:20.305765 20059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:20.296303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.2:43395)
0620 18:33:20.296892 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:20.296908 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:20.297017 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:20.297321 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:20.297325 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:20.297769 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:20.297934 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:20.297942 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:20.299113 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:20.299117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:20.300080 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:20.300093 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:20.300747 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:20.300752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:20.300998 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:20.301667 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:20.301686 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:20.305462 (+ 3776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":347,"threads_started":1}
W20260620 18:33:20.305990 20060 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:20.296367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:43395 (local address 127.10.131.62:35671)
0620 18:33:20.296827 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:20.296832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:20.296976 (+ 144us) server_negotiation.cc:407] Connection header received
0620 18:33:20.297107 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:20.297111 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:20.297171 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:20.297295 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:20.298054 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:20.298981 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:20.300211 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:20.300609 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:20.300702 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:20.305692 (+ 4990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.2:43395: BlockingRecv error: recv got EOF from 127.10.131.2:43395 (error 108)
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":248,"threads_started":1}
Jun 20 18:33:20 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:20 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:20.623955 20061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:20.614880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.1:45795)
0620 18:33:20.615332 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:20.615352 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:20.615461 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:20.615768 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:20.615771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:20.616154 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:20.616293 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:20.616301 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:20.617511 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:20.617515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:20.618627 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:20.618638 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:20.619442 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:20.619451 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:20.619698 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:20.620464 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:20.620481 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:20.623715 (+ 3234us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":267,"threads_started":1}
W20260620 18:33:20.624266 20060 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:20.615008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:45795 (local address 127.10.131.62:35671)
0620 18:33:20.615245 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:20.615251 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:20.615431 (+ 180us) server_negotiation.cc:407] Connection header received
0620 18:33:20.615559 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:20.615563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:20.615624 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:20.615729 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:20.616457 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:20.617369 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:20.618773 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:20.619280 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:20.619433 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:20.623978 (+ 4545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.1:45795: BlockingRecv error: recv got EOF from 127.10.131.1:45795 (error 108)
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":101}
Jun 20 18:33:20 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.3@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:20 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.3@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:20.628708 20062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:20.619136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.3:38789)
0620 18:33:20.620122 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:20.620144 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:20.620285 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:20.620611 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:20.620616 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:20.621066 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:20.621223 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:20.621232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:20.622282 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:20.622288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:20.623416 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:20.623434 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:20.624504 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:20.624510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:20.624661 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:20.625260 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:20.625276 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:20.628324 (+ 3048us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":883,"thread_start_us":763,"threads_started":1}
W20260620 18:33:20.628793 20063 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:20.619210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:38789 (local address 127.10.131.62:35671)
0620 18:33:20.619991 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:20.619997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:20.620232 (+ 235us) server_negotiation.cc:407] Connection header received
0620 18:33:20.620402 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:20.620408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:20.620466 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:20.620568 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:20.621373 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:20.622146 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:20.623593 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:20.624240 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:20.624373 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:20.628544 (+ 4171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.3:38789: BlockingRecv error: recv got EOF from 127.10.131.3:38789 (error 108)
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":654,"thread_start_us":525,"threads_started":1}
Jun 20 18:33:21 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.2@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:21 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.2@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:21.317865 20064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:21.306709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.2:47443)
0620 18:33:21.307433 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:21.307457 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:21.307597 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:21.308180 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:21.308185 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:21.308572 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:21.308711 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:21.308719 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:21.310371 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:21.310375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:21.311438 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:21.311450 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:21.312510 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:21.312514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:21.312651 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:21.313337 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:21.313355 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:21.317531 (+ 4176us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":422,"threads_started":1}
W20260620 18:33:21.318162 20065 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:21.307171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:47443 (local address 127.10.131.62:35671)
0620 18:33:21.307723 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:21.307731 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:21.307749 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:33:21.307798 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:21.307808 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:21.307864 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:21.307986 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:21.308884 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:21.310212 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:21.311573 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:21.312351 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:21.312487 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:21.317834 (+ 5347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.2:47443: BlockingRecv error: recv got EOF from 127.10.131.2:47443 (error 108)
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":297,"threads_started":1}
Jun 20 18:33:21 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:21 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.1@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:21.635210 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:21.625181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.1:56641)
0620 18:33:21.625715 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:21.625734 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:21.625837 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:21.626103 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:21.626106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:21.626476 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:21.626644 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:21.626653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:21.627669 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:21.627672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:21.628556 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:21.628566 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:21.629376 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:21.629383 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:21.629598 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:21.630787 (+ 1189us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:21.630808 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:21.634820 (+ 4012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":300,"threads_started":1}
W20260620 18:33:21.635296 20065 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:21.625267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:56641 (local address 127.10.131.62:35671)
0620 18:33:21.625419 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:21.625423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:21.625799 (+ 376us) server_negotiation.cc:407] Connection header received
0620 18:33:21.625915 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:21.625918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:21.625988 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:21.626064 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:21.626763 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:21.627551 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:21.628685 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:21.629221 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:21.629381 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:21.635031 (+ 5650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.1:56641: BlockingRecv error: recv got EOF from 127.10.131.1:56641 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 18:33:21 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.3@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:33:21 dist-test-slave-cbn3 krb5kdc[18995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.131.3@KRBTEST.COM for oryx/127.10.131.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:33:21.639117 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:21.629700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35671 (local address 127.10.131.3:49949)
0620 18:33:21.630514 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:21.630532 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:21.630657 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:21.631003 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:21.631009 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:21.631516 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:21.631691 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:21.631703 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:21.632781 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:21.632788 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:21.634020 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:21.634036 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:21.634985 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:21.634991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:21.635163 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:21.635793 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:21.635810 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:21.638900 (+ 3090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.131.62:35671: Server oryx/127.10.131.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":591,"threads_started":1}
W20260620 18:33:21.639679 20068 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:21.629748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:49949 (local address 127.10.131.62:35671)
0620 18:33:21.630396 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:21.630401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:21.630653 (+ 252us) server_negotiation.cc:407] Connection header received
0620 18:33:21.630773 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:21.630778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:21.630843 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:21.630960 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:21.631834 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:21.632650 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:21.634204 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:21.634821 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:21.634967 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:21.639196 (+ 4229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.131.3:49949: BlockingRecv error: recv got EOF from 127.10.131.3:49949 (error 108)
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":290,"threads_started":1}
I20260620 18:33:22.225111 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 19582
I20260620 18:33:22.238020 20070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:22.226305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49330 (local address 127.10.131.1:37579)
0620 18:33:22.226917 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:22.226924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:22.226949 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:33:22.227010 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:22.227016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:22.227241 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:22.227392 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:22.228743 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:22.229755 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:22.231005 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:22.231535 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:22.231630 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:22.234145 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:22.234159 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:22.234180 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:33:22.234264 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:22.237154 (+ 2890us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.10.131.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 18:33:22.237554 (+ 400us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49330: Request ticket server kudu/127.10.131.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":325,"threads_started":1}
W20260620 18:33:22.237933 10764 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 19582: Not authorized: Client connection negotiation failed: client connection to 127.10.131.1:37579: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.10.131.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 18:33:22.238137 20070 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49330: Request ticket server kudu/127.10.131.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 18:33:22.238206 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 19582
I20260620 18:33:22.255067 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 19733
I20260620 18:33:22.267726 20071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:22.256106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48838 (local address 127.10.131.2:35765)
0620 18:33:22.256708 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:22.256715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:22.256741 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:33:22.256806 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:22.256813 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:22.257133 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:22.257310 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:22.259161 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:22.260217 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:22.261551 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:22.262156 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:22.262255 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:22.264569 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:22.264577 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:22.264590 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:33:22.264632 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:22.267204 (+ 2572us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.10.131.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 18:33:22.267495 (+ 291us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48838: Request ticket server kudu/127.10.131.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":321,"threads_started":1}
W20260620 18:33:22.267865 20071 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48838: Request ticket server kudu/127.10.131.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 18:33:22.267846 10764 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 19733: Not authorized: Client connection negotiation failed: client connection to 127.10.131.2:35765: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.10.131.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 18:33:22.267994 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 19733
I20260620 18:33:22.284930 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 19866
W20260620 18:33:22.297246 10764 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 19866: Not authorized: Client connection negotiation failed: client connection to 127.10.131.3:40125: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.10.131.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 18:33:22.297420 20072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:22.286095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33470 (local address 127.10.131.3:40125)
0620 18:33:22.286626 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:22.286633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:22.286659 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:33:22.286722 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:22.286728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:22.286954 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:22.287134 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:22.288988 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:22.289929 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:22.291022 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:22.291557 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:22.291655 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:22.294117 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:22.294126 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:22.294140 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:33:22.294180 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:22.296575 (+ 2395us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.10.131.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 18:33:22.297003 (+ 428us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33470: Request ticket server kudu/127.10.131.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":258,"threads_started":1}
I20260620 18:33:22.297441 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 19866
I20260620 18:33:22.313998 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 19513
I20260620 18:33:22.315248 19575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:33:22.428018 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 19513
2026-06-20T18:33:22Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21187 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T18:33:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:33:22Z Disabled control of system clock
I20260620 18:33:22.502346 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:39009
--webserver_interface=127.10.131.62
--webserver_port=0
--builtin_ntp_servers=127.10.131.20:43511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:39009
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 18:33:22.825729 20077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:33:22.826010 20077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:33:22.826088 20077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:33:22.833314 20077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:33:22.833416 20077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:33:22.833453 20077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:33:22.833509 20077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:33:22.833559 20077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:33:22.843253 20077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:43511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:39009
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:39009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:33:22.844519 20077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:33:22.846272 20077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:33:22.858290 20085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:22.858331 20083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:22.858399 20082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:33:22.858743 20077 server_base.cc:1034] running on GCE node
I20260620 18:33:22.859530 20077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:33:22.860891 20077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:33:22.862103 20077 hybrid_clock.cc:648] HybridClock initialized: now 1781980402862054 us; error 43 us; skew 500 ppm
I20260620 18:33:22.862459 20077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:33:22.864746 20077 webserver.cc:460] Webserver started at http://127.10.131.62:32795/ using document root <none> and password file <none>
I20260620 18:33:22.865458 20077 fs_manager.cc:359] Metadata directory not provided
I20260620 18:33:22.865592 20077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:33:22.865921 20077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:33:22.868814 20077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "dca4df7628a14a4a8f82bbb8efd6ebe7"
format_stamp: "Formatted at 2026-06-20 18:33:22 on dist-test-slave-cbn3"
server_key: "320af796af4ff8538752fe3b40ebb29d"
server_key_iv: "31b018fdb9f67c0678d2e959f5c145c2"
server_key_version: "encryptionkey@0"
I20260620 18:33:22.869680 20077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "dca4df7628a14a4a8f82bbb8efd6ebe7"
format_stamp: "Formatted at 2026-06-20 18:33:22 on dist-test-slave-cbn3"
server_key: "320af796af4ff8538752fe3b40ebb29d"
server_key_iv: "31b018fdb9f67c0678d2e959f5c145c2"
server_key_version: "encryptionkey@0"
I20260620 18:33:22.874713 20077 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 18:33:22.878182 20091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:22.879575 20077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:33:22.879771 20077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "dca4df7628a14a4a8f82bbb8efd6ebe7"
format_stamp: "Formatted at 2026-06-20 18:33:22 on dist-test-slave-cbn3"
server_key: "320af796af4ff8538752fe3b40ebb29d"
server_key_iv: "31b018fdb9f67c0678d2e959f5c145c2"
server_key_version: "encryptionkey@0"
I20260620 18:33:22.879987 20077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:33:22.925482 20077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:33:22.926366 20077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:33:22.926674 20077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:33:22.945094 20077 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:39009
I20260620 18:33:22.945102 20142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:39009 every 8 connection(s)
I20260620 18:33:22.946471 20077 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:33:22.950160 20143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:33:22.953702 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 20077
I20260620 18:33:22.953891 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:33:22.954739 10764 external_mini_cluster.cc:1428] Setting key 1820ddbc8565d279ad78d4116ac198b7
I20260620 18:33:22.962633 20143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dca4df7628a14a4a8f82bbb8efd6ebe7: Bootstrap starting.
I20260620 18:33:22.965907 20143 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dca4df7628a14a4a8f82bbb8efd6ebe7: Neither blocks nor log segments found. Creating new log.
I20260620 18:33:22.966941 20143 log.cc:824] T 00000000000000000000000000000000 P dca4df7628a14a4a8f82bbb8efd6ebe7: Log is configured to *not* fsync() on all Append() calls
I20260620 18:33:22.968128 20146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:22.956794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53166 (local address 127.10.131.62:39009)
0620 18:33:22.957723 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:22.957734 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:22.957773 (+ 39us) server_negotiation.cc:407] Connection header received
0620 18:33:22.959296 (+ 1523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:22.959316 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:22.959767 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:22.960168 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:22.960739 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:22.962021 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:22.963395 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:22.964385 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:22.964619 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:22.965845 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:22.965853 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:22.965867 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:33:22.966456 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:22.966534 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:33:22.966554 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:22.966681 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:22.966901 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 18:33:22.967232 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":295,"threads_started":1}
I20260620 18:33:22.969816 20143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dca4df7628a14a4a8f82bbb8efd6ebe7: No bootstrap required, opened a new log
I20260620 18:33:22.977355 20143 raft_consensus.cc:348] T 00000000000000000000000000000000 P dca4df7628a14a4a8f82bbb8efd6ebe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca4df7628a14a4a8f82bbb8efd6ebe7" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 39009 } }
I20260620 18:33:22.977726 20143 raft_consensus.cc:374] T 00000000000000000000000000000000 P dca4df7628a14a4a8f82bbb8efd6ebe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:33:22.977842 20143 raft_consensus.cc:729] T 00000000000000000000000000000000 P dca4df7628a14a4a8f82bbb8efd6ebe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca4df7628a14a4a8f82bbb8efd6ebe7, State: Initialized, Role: FOLLOWER
I20260620 18:33:22.978390 20143 consensus_queue.cc:260] T 00000000000000000000000000000000 P dca4df7628a14a4a8f82bbb8efd6ebe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca4df7628a14a4a8f82bbb8efd6ebe7" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 39009 } }
I20260620 18:33:22.978621 20143 raft_consensus.cc:388] T 00000000000000000000000000000000 P dca4df7628a14a4a8f82bbb8efd6ebe7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:33:22.978752 20143 raft_consensus.cc:482] T 00000000000000000000000000000000 P dca4df7628a14a4a8f82bbb8efd6ebe7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:33:22.978909 20143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dca4df7628a14a4a8f82bbb8efd6ebe7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:33:22.981142 20143 raft_consensus.cc:504] T 00000000000000000000000000000000 P dca4df7628a14a4a8f82bbb8efd6ebe7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca4df7628a14a4a8f82bbb8efd6ebe7" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 39009 } }
I20260620 18:33:22.981696 20143 leader_election.cc:302] T 00000000000000000000000000000000 P dca4df7628a14a4a8f82bbb8efd6ebe7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dca4df7628a14a4a8f82bbb8efd6ebe7; no voters:
I20260620 18:33:22.982391 20143 leader_election.cc:288] T 00000000000000000000000000000000 P dca4df7628a14a4a8f82bbb8efd6ebe7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:33:22.982520 20148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dca4df7628a14a4a8f82bbb8efd6ebe7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:33:22.983309 20148 raft_consensus.cc:686] T 00000000000000000000000000000000 P dca4df7628a14a4a8f82bbb8efd6ebe7 [term 1 LEADER]: Becoming Leader. State: Replica: dca4df7628a14a4a8f82bbb8efd6ebe7, State: Running, Role: LEADER
I20260620 18:33:22.983822 20148 consensus_queue.cc:237] T 00000000000000000000000000000000 P dca4df7628a14a4a8f82bbb8efd6ebe7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca4df7628a14a4a8f82bbb8efd6ebe7" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 39009 } }
I20260620 18:33:22.984817 20143 sys_catalog.cc:564] T 00000000000000000000000000000000 P dca4df7628a14a4a8f82bbb8efd6ebe7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:33:22.986892 20149 sys_catalog.cc:455] T 00000000000000000000000000000000 P dca4df7628a14a4a8f82bbb8efd6ebe7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dca4df7628a14a4a8f82bbb8efd6ebe7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca4df7628a14a4a8f82bbb8efd6ebe7" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 39009 } } }
I20260620 18:33:22.987282 20149 sys_catalog.cc:458] T 00000000000000000000000000000000 P dca4df7628a14a4a8f82bbb8efd6ebe7 [sys.catalog]: This master's current role is: LEADER
I20260620 18:33:22.987771 20150 sys_catalog.cc:455] T 00000000000000000000000000000000 P dca4df7628a14a4a8f82bbb8efd6ebe7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dca4df7628a14a4a8f82bbb8efd6ebe7. Latest consensus state: current_term: 1 leader_uuid: "dca4df7628a14a4a8f82bbb8efd6ebe7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca4df7628a14a4a8f82bbb8efd6ebe7" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 39009 } } }
I20260620 18:33:22.988034 20150 sys_catalog.cc:458] T 00000000000000000000000000000000 P dca4df7628a14a4a8f82bbb8efd6ebe7 [sys.catalog]: This master's current role is: LEADER
W20260620 18:33:22.995903 20164 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dca4df7628a14a4a8f82bbb8efd6ebe7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:33:22.996009 20164 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:33:22.996418 20155 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:33:22.998056 20155 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:33:23.006342 20155 catalog_manager.cc:1347] Generated new cluster ID: aac541f79e1a4d29a53775a6e592d642
I20260620 18:33:23.006464 20155 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:33:23.021487 20155 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:33:23.022727 20155 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:33:23.039119 20155 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dca4df7628a14a4a8f82bbb8efd6ebe7: Generated new TSK 0
I20260620 18:33:23.040212 20155 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:33:23.056039 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.1:0
--local_ip_for_outbound_sockets=127.10.131.1
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:39009
--builtin_ntp_servers=127.10.131.20:43511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 18:33:23.384608 20167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:33:23.385030 20167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:33:23.385179 20167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:33:23.392941 20167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:33:23.393038 20167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:33:23.393151 20167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.1
I20260620 18:33:23.402195 20167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:43511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:39009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.1
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:33:23.403465 20167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:33:23.405176 20167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:33:23.418388 20173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:23.418450 20175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:23.418479 20172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:33:23.419157 20167 server_base.cc:1034] running on GCE node
I20260620 18:33:23.419709 20167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:33:23.420604 20167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:33:23.421793 20167 hybrid_clock.cc:648] HybridClock initialized: now 1781980403421725 us; error 59 us; skew 500 ppm
I20260620 18:33:23.422060 20167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:33:23.427336 20167 webserver.cc:460] Webserver started at http://127.10.131.1:43605/ using document root <none> and password file <none>
I20260620 18:33:23.427976 20167 fs_manager.cc:359] Metadata directory not provided
I20260620 18:33:23.428058 20167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:33:23.428328 20167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:33:23.431140 20167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/data/instance:
uuid: "1ac1621b85f44bc6b34b858a9e54f3de"
format_stamp: "Formatted at 2026-06-20 18:33:23 on dist-test-slave-cbn3"
server_key: "7cbdd51aaae64bee7e755ce70f7c398c"
server_key_iv: "5721e52cc6d9bb67229c3678dc7c7ea3"
server_key_version: "encryptionkey@0"
I20260620 18:33:23.431869 20167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance:
uuid: "1ac1621b85f44bc6b34b858a9e54f3de"
format_stamp: "Formatted at 2026-06-20 18:33:23 on dist-test-slave-cbn3"
server_key: "7cbdd51aaae64bee7e755ce70f7c398c"
server_key_iv: "5721e52cc6d9bb67229c3678dc7c7ea3"
server_key_version: "encryptionkey@0"
I20260620 18:33:23.436945 20167 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260620 18:33:23.440184 20181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:23.441548 20167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:33:23.441677 20167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/wal
uuid: "1ac1621b85f44bc6b34b858a9e54f3de"
format_stamp: "Formatted at 2026-06-20 18:33:23 on dist-test-slave-cbn3"
server_key: "7cbdd51aaae64bee7e755ce70f7c398c"
server_key_iv: "5721e52cc6d9bb67229c3678dc7c7ea3"
server_key_version: "encryptionkey@0"
I20260620 18:33:23.441848 20167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:33:23.465034 20167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:33:23.465873 20167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:33:23.466120 20167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:33:23.466979 20167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:33:23.468575 20167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:33:23.468693 20167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:23.468824 20167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:33:23.468942 20167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 18:33:23.502199 20167 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.1:40641
I20260620 18:33:23.502285 20293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.1:40641 every 8 connection(s)
I20260620 18:33:23.503475 20167 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
I20260620 18:33:23.508708 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 20167
I20260620 18:33:23.508924 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance
I20260620 18:33:23.509531 10764 external_mini_cluster.cc:1428] Setting key 5697ff3080cc61c4545f76cd255613a6
I20260620 18:33:23.512003 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.2:0
--local_ip_for_outbound_sockets=127.10.131.2
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:39009
--builtin_ntp_servers=127.10.131.20:43511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 18:33:23.516112 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:23.506080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:43777 (local address 127.10.131.62:39009)
0620 18:33:23.506520 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:23.506531 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:23.507576 (+ 1045us) server_negotiation.cc:407] Connection header received
0620 18:33:23.509578 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:23.509584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:23.509671 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:23.509790 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:23.510952 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:23.511752 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:23.513417 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:23.513808 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:23.513940 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:23.515292 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:23.515297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:23.515300 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:33:23.515317 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:23.515357 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:33:23.515369 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:23.515465 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:23.515683 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 18:33:23.515777 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":242,"threads_started":1}
I20260620 18:33:23.517198 20297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:23.506553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:39009 (local address 127.10.131.1:43777)
0620 18:33:23.507404 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:23.507444 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:23.509300 (+ 1856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:23.510015 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:23.510031 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:23.510123 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:23.510790 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:23.510806 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:23.511910 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:23.511914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:23.513275 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:23.513288 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:23.514037 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:23.514045 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:23.514270 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:23.514960 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:33:23.514962 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:23.515001 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:33:23.515093 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:33:23.515463 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:33:23.515475 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:33:23.515485 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 18:33:23.515675 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 18:33:23.516096 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":254,"threads_started":1}
I20260620 18:33:23.518743 20294 heartbeater.cc:347] Connected to a master server at 127.10.131.62:39009
I20260620 18:33:23.519054 20294 heartbeater.cc:464] Registering TS with master...
I20260620 18:33:23.519791 20294 heartbeater.cc:511] Master 127.10.131.62:39009 requested a full tablet report, sending...
I20260620 18:33:23.522019 20108 ts_manager.cc:194] Registered new tserver with Master: 1ac1621b85f44bc6b34b858a9e54f3de (127.10.131.1:40641)
I20260620 18:33:23.524220 20108 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.131.1:43777
W20260620 18:33:23.840461 20298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:33:23.840728 20298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:33:23.840811 20298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:33:23.848266 20298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:33:23.848362 20298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:33:23.848467 20298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.2
I20260620 18:33:23.857465 20298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:43511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:39009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.2
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:33:23.858655 20298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:33:23.860348 20298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:33:23.873293 20306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:23.873359 20303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:23.873360 20304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:33:23.874143 20298 server_base.cc:1034] running on GCE node
I20260620 18:33:23.874786 20298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:33:23.875705 20298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:33:23.876958 20298 hybrid_clock.cc:648] HybridClock initialized: now 1781980403876908 us; error 45 us; skew 500 ppm
I20260620 18:33:23.877332 20298 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:33:23.879622 20298 webserver.cc:460] Webserver started at http://127.10.131.2:37565/ using document root <none> and password file <none>
I20260620 18:33:23.880286 20298 fs_manager.cc:359] Metadata directory not provided
I20260620 18:33:23.880373 20298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:33:23.880640 20298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:33:23.883636 20298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/data/instance:
uuid: "3cc595938b12468fa694d1950476264c"
format_stamp: "Formatted at 2026-06-20 18:33:23 on dist-test-slave-cbn3"
server_key: "6041dcb6d57bbc4af9a3a4e70ff2e6c9"
server_key_iv: "3a8739bf8768dd5b89891d39d429bbbc"
server_key_version: "encryptionkey@0"
I20260620 18:33:23.884524 20298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance:
uuid: "3cc595938b12468fa694d1950476264c"
format_stamp: "Formatted at 2026-06-20 18:33:23 on dist-test-slave-cbn3"
server_key: "6041dcb6d57bbc4af9a3a4e70ff2e6c9"
server_key_iv: "3a8739bf8768dd5b89891d39d429bbbc"
server_key_version: "encryptionkey@0"
I20260620 18:33:23.889930 20298 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 18:33:23.893646 20312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:23.895128 20298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:33:23.895299 20298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/wal
uuid: "3cc595938b12468fa694d1950476264c"
format_stamp: "Formatted at 2026-06-20 18:33:23 on dist-test-slave-cbn3"
server_key: "6041dcb6d57bbc4af9a3a4e70ff2e6c9"
server_key_iv: "3a8739bf8768dd5b89891d39d429bbbc"
server_key_version: "encryptionkey@0"
I20260620 18:33:23.895525 20298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:33:23.905992 20298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:33:23.906780 20298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:33:23.907079 20298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:33:23.908028 20298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:33:23.909488 20298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:33:23.909591 20298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:23.909725 20298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:33:23.909798 20298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:23.942209 20298 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.2:42817
I20260620 18:33:23.942293 20424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.2:42817 every 8 connection(s)
I20260620 18:33:23.943508 20298 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
I20260620 18:33:23.945856 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 20298
I20260620 18:33:23.946045 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance
I20260620 18:33:23.946710 10764 external_mini_cluster.cc:1428] Setting key 4a6bf69cff519660d3898ecd25d8cce3
I20260620 18:33:23.949541 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.3:0
--local_ip_for_outbound_sockets=127.10.131.3
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:39009
--builtin_ntp_servers=127.10.131.20:43511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 18:33:23.959585 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:23.946420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:49487 (local address 127.10.131.62:39009)
0620 18:33:23.946608 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:23.946613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:23.950100 (+ 3487us) server_negotiation.cc:407] Connection header received
0620 18:33:23.952512 (+ 2412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:23.952520 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:23.952605 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:23.952714 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:23.954079 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:23.955292 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:23.956823 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:23.957383 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:23.957504 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:23.958765 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:23.958771 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:23.958774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:33:23.958791 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:23.958820 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:33:23.958831 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:23.958926 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:23.959185 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 18:33:23.959279 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:33:23.960646 20427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:23.947593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:39009 (local address 127.10.131.2:49487)
0620 18:33:23.949926 (+ 2333us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:23.949979 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:23.952216 (+ 2237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:23.952976 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:23.952987 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:23.953104 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:23.953890 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:23.953921 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:23.955482 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:23.955486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:23.956686 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:23.956698 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:23.957583 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:23.957588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:23.957780 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:23.958459 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:33:23.958462 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:23.958498 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:33:23.958572 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:33:23.958924 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:33:23.958935 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:33:23.958945 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 18:33:23.959172 (+ 227us) client_negotiation.cc:240] Negotiation successful
0620 18:33:23.959550 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":315,"threads_started":1}
I20260620 18:33:23.961977 20425 heartbeater.cc:347] Connected to a master server at 127.10.131.62:39009
I20260620 18:33:23.962293 20425 heartbeater.cc:464] Registering TS with master...
I20260620 18:33:23.963016 20425 heartbeater.cc:511] Master 127.10.131.62:39009 requested a full tablet report, sending...
I20260620 18:33:23.964360 20108 ts_manager.cc:194] Registered new tserver with Master: 3cc595938b12468fa694d1950476264c (127.10.131.2:42817)
I20260620 18:33:23.965231 20108 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.131.2:49487
W20260620 18:33:24.273275 20428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:33:24.273609 20428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:33:24.273737 20428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:33:24.281561 20428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:33:24.281665 20428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:33:24.281867 20428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.3
I20260620 18:33:24.291621 20428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:43511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:39009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.3
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:33:24.292994 20428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:33:24.294759 20428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:33:24.308024 20433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:24.308049 20434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:24.307987 20436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:33:24.308876 20428 server_base.cc:1034] running on GCE node
I20260620 18:33:24.309594 20428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:33:24.310629 20428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:33:24.311873 20428 hybrid_clock.cc:648] HybridClock initialized: now 1781980404311803 us; error 56 us; skew 500 ppm
I20260620 18:33:24.312207 20428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:33:24.314704 20428 webserver.cc:460] Webserver started at http://127.10.131.3:45155/ using document root <none> and password file <none>
I20260620 18:33:24.315423 20428 fs_manager.cc:359] Metadata directory not provided
I20260620 18:33:24.315558 20428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:33:24.315896 20428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:33:24.318763 20428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/data/instance:
uuid: "11ea0beb1eb64b79b1fa120794836b66"
format_stamp: "Formatted at 2026-06-20 18:33:24 on dist-test-slave-cbn3"
server_key: "4eca57fdbd6462d955daca7eb4b6d5fc"
server_key_iv: "650722b9f6b532f233367c3b0d02c528"
server_key_version: "encryptionkey@0"
I20260620 18:33:24.319566 20428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance:
uuid: "11ea0beb1eb64b79b1fa120794836b66"
format_stamp: "Formatted at 2026-06-20 18:33:24 on dist-test-slave-cbn3"
server_key: "4eca57fdbd6462d955daca7eb4b6d5fc"
server_key_iv: "650722b9f6b532f233367c3b0d02c528"
server_key_version: "encryptionkey@0"
I20260620 18:33:24.324725 20428 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:33:24.328054 20442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:24.329583 20428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:33:24.329761 20428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/wal
uuid: "11ea0beb1eb64b79b1fa120794836b66"
format_stamp: "Formatted at 2026-06-20 18:33:24 on dist-test-slave-cbn3"
server_key: "4eca57fdbd6462d955daca7eb4b6d5fc"
server_key_iv: "650722b9f6b532f233367c3b0d02c528"
server_key_version: "encryptionkey@0"
I20260620 18:33:24.329977 20428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:33:24.351114 20428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:33:24.351981 20428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:33:24.352306 20428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:33:24.353327 20428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:33:24.354732 20428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:33:24.354851 20428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:24.354984 20428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:33:24.355070 20428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:24.387940 20428 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.3:38767
I20260620 18:33:24.388034 20554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.3:38767 every 8 connection(s)
I20260620 18:33:24.389387 20428 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
I20260620 18:33:24.394200 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 20428
I20260620 18:33:24.394384 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance
I20260620 18:33:24.394850 10764 external_mini_cluster.cc:1428] Setting key 64e07dd7974e48f37ff0e0549e9cffd6
I20260620 18:33:24.402159 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:24.392007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:36413 (local address 127.10.131.62:39009)
0620 18:33:24.392180 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:24.392184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:24.393778 (+ 1594us) server_negotiation.cc:407] Connection header received
0620 18:33:24.395626 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:24.395631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:24.395720 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:24.395829 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:24.397217 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:24.398071 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:24.399671 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:24.400084 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:24.400172 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:24.401447 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:24.401452 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:24.401455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:33:24.401471 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:24.401500 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:33:24.401512 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:24.401588 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:24.401854 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 18:33:24.401928 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:33:24.403316 20557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:24.392562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:39009 (local address 127.10.131.3:36413)
0620 18:33:24.393598 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:24.393642 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:24.395359 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:24.396093 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:24.396105 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:24.396229 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:24.397056 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:24.397076 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:24.398197 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:24.398202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:24.399548 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:24.399560 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:24.400289 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:24.400294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:24.400485 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:24.401135 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:33:24.401138 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:24.401177 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:33:24.401281 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:33:24.401635 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:33:24.401645 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:33:24.401652 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 18:33:24.401828 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 18:33:24.402175 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":351,"threads_started":1}
I20260620 18:33:24.404731 20555 heartbeater.cc:347] Connected to a master server at 127.10.131.62:39009
I20260620 18:33:24.405079 20555 heartbeater.cc:464] Registering TS with master...
I20260620 18:33:24.405826 20555 heartbeater.cc:511] Master 127.10.131.62:39009 requested a full tablet report, sending...
I20260620 18:33:24.407219 20108 ts_manager.cc:194] Registered new tserver with Master: 11ea0beb1eb64b79b1fa120794836b66 (127.10.131.3:38767)
I20260620 18:33:24.407972 20108 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.131.3:36413
I20260620 18:33:24.410567 10764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 18:33:24.416795 20296 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:24.415051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53176 (local address 127.10.131.62:39009)
0620 18:33:24.415239 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:24.415244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:24.415813 (+ 569us) server_negotiation.cc:407] Connection header received
0620 18:33:24.415957 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:24.415962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:24.416046 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:24.416138 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:24.416459 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53176: BlockingRecv error: recv got EOF from 127.0.0.1:53176 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:33:24.417662 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 20167
I20260620 18:33:24.425510 20564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:24.418622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37332 (local address 127.10.131.1:40641)
0620 18:33:24.419177 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:24.419187 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:24.419217 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:33:24.419272 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:24.419278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:24.419494 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:24.419618 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:24.420023 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:24.421136 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:24.422346 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:24.422867 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:24.422973 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:24.424029 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:24.424038 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:24.424051 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:33:24.424693 (+ 642us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:24.424754 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:33:24.424775 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:24.424958 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:24.425074 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 18:33:24.425191 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":294,"threads_started":1}
I20260620 18:33:24.426760 20289 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:33:24.508249 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 20167
I20260620 18:33:24.521870 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 20298
I20260620 18:33:24.529809 20566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:24.522754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45386 (local address 127.10.131.2:42817)
0620 18:33:24.523336 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:24.523343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:24.523371 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:33:24.523432 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:24.523438 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:24.523651 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:24.523780 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:24.524227 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:24.525315 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:24.526595 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:24.527149 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:24.527253 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:24.528382 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:24.528389 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:24.528402 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:33:24.529097 (+ 695us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:24.529144 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:33:24.529164 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:24.529252 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:24.529410 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 18:33:24.529519 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":288,"threads_started":1}
I20260620 18:33:24.531056 20420 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:33:24.613610 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 20298
I20260620 18:33:24.630195 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 20428
I20260620 18:33:24.638190 20568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:24.631144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45860 (local address 127.10.131.3:38767)
0620 18:33:24.631750 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:24.631757 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:24.631785 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:33:24.631848 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:24.631855 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:24.632064 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:24.632188 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:24.632617 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:24.633777 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:24.635062 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:24.635605 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:24.635702 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:24.636703 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:24.636711 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:24.636725 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:33:24.637395 (+ 670us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:24.637440 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:33:24.637459 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:24.637545 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:24.637693 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 18:33:24.637824 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":311,"threads_started":1}
I20260620 18:33:24.639472 20550 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:33:24.720917 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 20428
I20260620 18:33:24.736522 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 20077
I20260620 18:33:24.737751 20138 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:33:24.839756 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 20077
2026-06-20T18:33:24Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2406 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T18:33:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:33:24Z Disabled control of system clock
I20260620 18:33:24.910903 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:41355
--webserver_interface=127.10.131.62
--webserver_port=0
--builtin_ntp_servers=127.10.131.20:38235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:41355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 18:33:25.238480 20574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:33:25.238780 20574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:33:25.238868 20574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:33:25.246140 20574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:33:25.246248 20574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:33:25.246286 20574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:33:25.246347 20574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:33:25.246418 20574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:33:25.255676 20574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:38235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:41355
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:41355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:33:25.256935 20574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:33:25.258636 20574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:33:25.270684 20579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:25.270628 20580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:25.270576 20582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:33:25.271553 20574 server_base.cc:1034] running on GCE node
I20260620 18:33:25.272356 20574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:33:25.273882 20574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:33:25.275115 20574 hybrid_clock.cc:648] HybridClock initialized: now 1781980405275070 us; error 37 us; skew 500 ppm
I20260620 18:33:25.275552 20574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:33:25.277936 20574 webserver.cc:460] Webserver started at http://127.10.131.62:36707/ using document root <none> and password file <none>
I20260620 18:33:25.278652 20574 fs_manager.cc:359] Metadata directory not provided
I20260620 18:33:25.278795 20574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:33:25.279134 20574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:33:25.282162 20574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "90c41a584c0948f7b22b8099f403a73b"
format_stamp: "Formatted at 2026-06-20 18:33:25 on dist-test-slave-cbn3"
server_key: "0557e9f4c7af564a9e26b084fa8704c9"
server_key_iv: "b79e3a55345d27582b92c781d01e1224"
server_key_version: "encryptionkey@0"
I20260620 18:33:25.282979 20574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "90c41a584c0948f7b22b8099f403a73b"
format_stamp: "Formatted at 2026-06-20 18:33:25 on dist-test-slave-cbn3"
server_key: "0557e9f4c7af564a9e26b084fa8704c9"
server_key_iv: "b79e3a55345d27582b92c781d01e1224"
server_key_version: "encryptionkey@0"
I20260620 18:33:25.287976 20574 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 18:33:25.291386 20588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:25.292634 20574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:33:25.292815 20574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "90c41a584c0948f7b22b8099f403a73b"
format_stamp: "Formatted at 2026-06-20 18:33:25 on dist-test-slave-cbn3"
server_key: "0557e9f4c7af564a9e26b084fa8704c9"
server_key_iv: "b79e3a55345d27582b92c781d01e1224"
server_key_version: "encryptionkey@0"
I20260620 18:33:25.293100 20574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:33:25.298559 20574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:33:25.299315 20574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:33:25.299618 20574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:33:25.317579 20574 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:41355
I20260620 18:33:25.317629 20639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:41355 every 8 connection(s)
I20260620 18:33:25.318980 20574 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:33:25.321780 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 20574
I20260620 18:33:25.322021 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:33:25.322897 10764 external_mini_cluster.cc:1428] Setting key 2f7dc3deed857c60b40c9aaed0ad2ee3
I20260620 18:33:25.323561 20640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:33:25.332106 20643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:25.325092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38572 (local address 127.10.131.62:41355)
0620 18:33:25.326124 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:25.326135 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:25.326175 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:33:25.328309 (+ 2134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:25.328331 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:25.328812 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:25.329308 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:25.330407 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:25.330413 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:25.330436 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:33:25.330450 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:25.330511 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:33:25.330527 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:25.330777 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:25.331006 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 18:33:25.331224 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":393,"threads_started":1}
I20260620 18:33:25.337816 20640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90c41a584c0948f7b22b8099f403a73b: Bootstrap starting.
I20260620 18:33:25.341475 20640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90c41a584c0948f7b22b8099f403a73b: Neither blocks nor log segments found. Creating new log.
I20260620 18:33:25.342558 20640 log.cc:824] T 00000000000000000000000000000000 P 90c41a584c0948f7b22b8099f403a73b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:33:25.345594 20640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90c41a584c0948f7b22b8099f403a73b: No bootstrap required, opened a new log
I20260620 18:33:25.353338 20640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90c41a584c0948f7b22b8099f403a73b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90c41a584c0948f7b22b8099f403a73b" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 41355 } }
I20260620 18:33:25.353809 20640 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90c41a584c0948f7b22b8099f403a73b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:33:25.353928 20640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90c41a584c0948f7b22b8099f403a73b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90c41a584c0948f7b22b8099f403a73b, State: Initialized, Role: FOLLOWER
I20260620 18:33:25.354590 20640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90c41a584c0948f7b22b8099f403a73b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90c41a584c0948f7b22b8099f403a73b" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 41355 } }
I20260620 18:33:25.354871 20640 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90c41a584c0948f7b22b8099f403a73b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:33:25.355008 20640 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90c41a584c0948f7b22b8099f403a73b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:33:25.355199 20640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90c41a584c0948f7b22b8099f403a73b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:33:25.357512 20640 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90c41a584c0948f7b22b8099f403a73b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90c41a584c0948f7b22b8099f403a73b" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 41355 } }
I20260620 18:33:25.358049 20640 leader_election.cc:302] T 00000000000000000000000000000000 P 90c41a584c0948f7b22b8099f403a73b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90c41a584c0948f7b22b8099f403a73b; no voters:
I20260620 18:33:25.358659 20640 leader_election.cc:288] T 00000000000000000000000000000000 P 90c41a584c0948f7b22b8099f403a73b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:33:25.358829 20645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90c41a584c0948f7b22b8099f403a73b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:33:25.359584 20645 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90c41a584c0948f7b22b8099f403a73b [term 1 LEADER]: Becoming Leader. State: Replica: 90c41a584c0948f7b22b8099f403a73b, State: Running, Role: LEADER
I20260620 18:33:25.360266 20645 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90c41a584c0948f7b22b8099f403a73b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90c41a584c0948f7b22b8099f403a73b" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 41355 } }
I20260620 18:33:25.360983 20640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90c41a584c0948f7b22b8099f403a73b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:33:25.363744 20646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90c41a584c0948f7b22b8099f403a73b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90c41a584c0948f7b22b8099f403a73b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90c41a584c0948f7b22b8099f403a73b" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 41355 } } }
I20260620 18:33:25.364112 20646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90c41a584c0948f7b22b8099f403a73b [sys.catalog]: This master's current role is: LEADER
I20260620 18:33:25.363893 20647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90c41a584c0948f7b22b8099f403a73b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90c41a584c0948f7b22b8099f403a73b. Latest consensus state: current_term: 1 leader_uuid: "90c41a584c0948f7b22b8099f403a73b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90c41a584c0948f7b22b8099f403a73b" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 41355 } } }
I20260620 18:33:25.364300 20647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90c41a584c0948f7b22b8099f403a73b [sys.catalog]: This master's current role is: LEADER
I20260620 18:33:25.369848 20654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:33:25.375258 20654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:33:25.383415 20654 catalog_manager.cc:1347] Generated new cluster ID: 2f339682a1274079a6d34194d877a0ae
I20260620 18:33:25.383535 20654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:33:25.396179 20654 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:33:25.396620 20654 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:33:25.404518 20654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90c41a584c0948f7b22b8099f403a73b: Generated new TSK 0
I20260620 18:33:25.405367 20654 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:33:25.419137 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.1:0
--local_ip_for_outbound_sockets=127.10.131.1
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:41355
--builtin_ntp_servers=127.10.131.20:38235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 18:33:25.753808 20664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:33:25.754096 20664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:33:25.754246 20664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:33:25.761667 20664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:33:25.761814 20664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:33:25.761996 20664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.1
I20260620 18:33:25.771490 20664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:38235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:41355
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.1
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:33:25.772807 20664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:33:25.774580 20664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:33:25.787883 20672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:25.787925 20670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:25.787964 20669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:33:25.790232 20664 server_base.cc:1034] running on GCE node
I20260620 18:33:25.790925 20664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:33:25.791890 20664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:33:25.793111 20664 hybrid_clock.cc:648] HybridClock initialized: now 1781980405793058 us; error 47 us; skew 500 ppm
I20260620 18:33:25.793429 20664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:33:25.795650 20664 webserver.cc:460] Webserver started at http://127.10.131.1:34091/ using document root <none> and password file <none>
I20260620 18:33:25.796394 20664 fs_manager.cc:359] Metadata directory not provided
I20260620 18:33:25.796527 20664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:33:25.796893 20664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:33:25.799836 20664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/data/instance:
uuid: "c8efec1e903c4478802f76d73e30e2e3"
format_stamp: "Formatted at 2026-06-20 18:33:25 on dist-test-slave-cbn3"
server_key: "c6b3084cc5c4f6f5301534b5d3f83761"
server_key_iv: "58e3d12b4f4f3cb96005a5acd21d21df"
server_key_version: "encryptionkey@0"
I20260620 18:33:25.800623 20664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance:
uuid: "c8efec1e903c4478802f76d73e30e2e3"
format_stamp: "Formatted at 2026-06-20 18:33:25 on dist-test-slave-cbn3"
server_key: "c6b3084cc5c4f6f5301534b5d3f83761"
server_key_iv: "58e3d12b4f4f3cb96005a5acd21d21df"
server_key_version: "encryptionkey@0"
I20260620 18:33:25.805785 20664 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260620 18:33:25.809214 20678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:25.810474 20664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 18:33:25.810650 20664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/wal
uuid: "c8efec1e903c4478802f76d73e30e2e3"
format_stamp: "Formatted at 2026-06-20 18:33:25 on dist-test-slave-cbn3"
server_key: "c6b3084cc5c4f6f5301534b5d3f83761"
server_key_iv: "58e3d12b4f4f3cb96005a5acd21d21df"
server_key_version: "encryptionkey@0"
I20260620 18:33:25.810868 20664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:33:25.816473 20664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:33:25.817350 20664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:33:25.817624 20664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:33:25.818516 20664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:33:25.820006 20664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:33:25.820124 20664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:25.820248 20664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:33:25.820335 20664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:25.853271 20664 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.1:33169
I20260620 18:33:25.853317 20790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.1:33169 every 8 connection(s)
I20260620 18:33:25.854732 20664 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
I20260620 18:33:25.861258 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 20664
I20260620 18:33:25.861474 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance
I20260620 18:33:25.862243 10764 external_mini_cluster.cc:1428] Setting key ec992266efeedcdf1a3f1e9ff9d21d4b
I20260620 18:33:25.863490 20793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:25.857381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:50033 (local address 127.10.131.62:41355)
0620 18:33:25.857834 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:25.857838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:25.858957 (+ 1119us) server_negotiation.cc:407] Connection header received
0620 18:33:25.860644 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:25.860649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:25.860723 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:25.860874 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:25.862590 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:25.862596 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:25.862601 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:33:25.862621 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:25.862669 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:33:25.862687 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:25.862811 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:25.863060 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 18:33:25.863132 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":258,"threads_started":1}
I20260620 18:33:25.864553 20794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:25.857934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:41355 (local address 127.10.131.1:50033)
0620 18:33:25.858806 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:25.858840 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:25.860361 (+ 1521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:25.861204 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:25.861213 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:25.861310 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:25.862218 (+ 908us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:33:25.862221 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:25.862259 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:33:25.862403 (+ 144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:33:25.862789 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:33:25.862796 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:33:25.862802 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 18:33:25.863137 (+ 335us) client_negotiation.cc:240] Negotiation successful
0620 18:33:25.863466 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":262,"threads_started":1}
I20260620 18:33:25.865190 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.2:0
--local_ip_for_outbound_sockets=127.10.131.2
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:41355
--builtin_ntp_servers=127.10.131.20:38235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 18:33:25.866489 20791 heartbeater.cc:347] Connected to a master server at 127.10.131.62:41355
I20260620 18:33:25.866798 20791 heartbeater.cc:464] Registering TS with master...
I20260620 18:33:25.867453 20791 heartbeater.cc:511] Master 127.10.131.62:41355 requested a full tablet report, sending...
I20260620 18:33:25.869704 20605 ts_manager.cc:194] Registered new tserver with Master: c8efec1e903c4478802f76d73e30e2e3 (127.10.131.1:33169)
W20260620 18:33:26.191421 20795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:33:26.191701 20795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:33:26.191789 20795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:33:26.199054 20795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:33:26.199152 20795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:33:26.199260 20795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.2
I20260620 18:33:26.208228 20795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:38235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:41355
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.2
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:33:26.209502 20795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:33:26.211194 20795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:33:26.224637 20800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:26.224731 20803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:26.224637 20801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:33:26.225612 20795 server_base.cc:1034] running on GCE node
I20260620 18:33:26.226220 20795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:33:26.227130 20795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:33:26.228322 20795 hybrid_clock.cc:648] HybridClock initialized: now 1781980406228283 us; error 35 us; skew 500 ppm
I20260620 18:33:26.228650 20795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:33:26.230952 20795 webserver.cc:460] Webserver started at http://127.10.131.2:42977/ using document root <none> and password file <none>
I20260620 18:33:26.231662 20795 fs_manager.cc:359] Metadata directory not provided
I20260620 18:33:26.231772 20795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:33:26.232115 20795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:33:26.235028 20795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/data/instance:
uuid: "79bf0611578c46e1842482e995c5ce4e"
format_stamp: "Formatted at 2026-06-20 18:33:26 on dist-test-slave-cbn3"
server_key: "82caeaa06870f3288fb678faff144c4f"
server_key_iv: "94c0e2768b58180676ca6605d0e5e081"
server_key_version: "encryptionkey@0"
I20260620 18:33:26.235816 20795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance:
uuid: "79bf0611578c46e1842482e995c5ce4e"
format_stamp: "Formatted at 2026-06-20 18:33:26 on dist-test-slave-cbn3"
server_key: "82caeaa06870f3288fb678faff144c4f"
server_key_iv: "94c0e2768b58180676ca6605d0e5e081"
server_key_version: "encryptionkey@0"
I20260620 18:33:26.240886 20795 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:33:26.244169 20809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:26.245536 20795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 18:33:26.245708 20795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/wal
uuid: "79bf0611578c46e1842482e995c5ce4e"
format_stamp: "Formatted at 2026-06-20 18:33:26 on dist-test-slave-cbn3"
server_key: "82caeaa06870f3288fb678faff144c4f"
server_key_iv: "94c0e2768b58180676ca6605d0e5e081"
server_key_version: "encryptionkey@0"
I20260620 18:33:26.245929 20795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:33:26.251472 20795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:33:26.252290 20795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:33:26.252562 20795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:33:26.253510 20795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:33:26.254899 20795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:33:26.255025 20795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:26.255151 20795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:33:26.255236 20795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:26.288640 20795 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.2:46503
I20260620 18:33:26.288713 20921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.2:46503 every 8 connection(s)
I20260620 18:33:26.290042 20795 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
I20260620 18:33:26.298262 20793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:26.292634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:58527 (local address 127.10.131.62:41355)
0620 18:33:26.292806 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:26.292811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:26.294235 (+ 1424us) server_negotiation.cc:407] Connection header received
0620 18:33:26.295919 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:26.295925 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:26.296015 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:26.296134 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:26.297498 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:26.297503 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:26.297508 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:33:26.297550 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:26.297580 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:33:26.297591 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:26.297719 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:26.297901 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 18:33:26.297969 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:33:26.299218 20924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:26.293217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:41355 (local address 127.10.131.2:58527)
0620 18:33:26.294073 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:26.294108 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:26.295655 (+ 1547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:26.296376 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:26.296384 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:26.296533 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:26.297196 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:33:26.297199 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:26.297235 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:33:26.297317 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:33:26.297690 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:33:26.297700 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:33:26.297707 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 18:33:26.297983 (+ 276us) client_negotiation.cc:240] Negotiation successful
0620 18:33:26.298230 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":266,"threads_started":1}
I20260620 18:33:26.299834 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 20795
I20260620 18:33:26.300040 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance
I20260620 18:33:26.300633 10764 external_mini_cluster.cc:1428] Setting key a8e0c08a425ad902a59c52d0d53e6665
I20260620 18:33:26.300781 20922 heartbeater.cc:347] Connected to a master server at 127.10.131.62:41355
I20260620 18:33:26.301121 20922 heartbeater.cc:464] Registering TS with master...
I20260620 18:33:26.301745 20922 heartbeater.cc:511] Master 127.10.131.62:41355 requested a full tablet report, sending...
I20260620 18:33:26.303179 20605 ts_manager.cc:194] Registered new tserver with Master: 79bf0611578c46e1842482e995c5ce4e (127.10.131.2:46503)
I20260620 18:33:26.303560 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.3:0
--local_ip_for_outbound_sockets=127.10.131.3
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:41355
--builtin_ntp_servers=127.10.131.20:38235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 18:33:26.632007 20925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:33:26.632316 20925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:33:26.632479 20925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:33:26.639662 20925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:33:26.639802 20925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:33:26.639986 20925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.3
I20260620 18:33:26.649086 20925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:38235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:41355
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.3
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:33:26.650467 20925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:33:26.652190 20925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:33:26.665333 20933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:26.665350 20930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:26.665755 20931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:33:26.666468 20925 server_base.cc:1034] running on GCE node
I20260620 18:33:26.667022 20925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:33:26.668044 20925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:33:26.669283 20925 hybrid_clock.cc:648] HybridClock initialized: now 1781980406669243 us; error 47 us; skew 500 ppm
I20260620 18:33:26.669607 20925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:33:26.672015 20925 webserver.cc:460] Webserver started at http://127.10.131.3:36551/ using document root <none> and password file <none>
I20260620 18:33:26.672721 20925 fs_manager.cc:359] Metadata directory not provided
I20260620 18:33:26.672849 20925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:33:26.673206 20925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:33:26.676328 20925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/data/instance:
uuid: "1ad3a62a30fd48a6aafde4d2da222b90"
format_stamp: "Formatted at 2026-06-20 18:33:26 on dist-test-slave-cbn3"
server_key: "bae1a4eb4677146cc7cd2a2576275473"
server_key_iv: "1a672e8f7e3b795b1117c34ded2c99b9"
server_key_version: "encryptionkey@0"
I20260620 18:33:26.677296 20925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance:
uuid: "1ad3a62a30fd48a6aafde4d2da222b90"
format_stamp: "Formatted at 2026-06-20 18:33:26 on dist-test-slave-cbn3"
server_key: "bae1a4eb4677146cc7cd2a2576275473"
server_key_iv: "1a672e8f7e3b795b1117c34ded2c99b9"
server_key_version: "encryptionkey@0"
I20260620 18:33:26.682639 20925 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:33:26.686299 20939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:26.687592 20925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:33:26.687781 20925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/wal
uuid: "1ad3a62a30fd48a6aafde4d2da222b90"
format_stamp: "Formatted at 2026-06-20 18:33:26 on dist-test-slave-cbn3"
server_key: "bae1a4eb4677146cc7cd2a2576275473"
server_key_iv: "1a672e8f7e3b795b1117c34ded2c99b9"
server_key_version: "encryptionkey@0"
I20260620 18:33:26.688012 20925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:33:26.693854 20925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:33:26.694687 20925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:33:26.694975 20925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:33:26.695894 20925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:33:26.697687 20925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:33:26.697815 20925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:26.697942 20925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:33:26.698071 20925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:26.732252 20925 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.3:38989
I20260620 18:33:26.732295 21051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.3:38989 every 8 connection(s)
I20260620 18:33:26.733704 20925 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
I20260620 18:33:26.735836 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 20925
I20260620 18:33:26.736030 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance
I20260620 18:33:26.736524 10764 external_mini_cluster.cc:1428] Setting key 90cb8ec16c5d3e46ede7000f5c0d7e59
I20260620 18:33:26.742489 20793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:26.736593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:54679 (local address 127.10.131.62:41355)
0620 18:33:26.736824 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:26.736831 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:26.738719 (+ 1888us) server_negotiation.cc:407] Connection header received
0620 18:33:26.740557 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:26.740560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:26.740610 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:26.740700 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:26.741910 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:26.741914 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:26.741917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:33:26.741931 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:26.741960 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:33:26.741972 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:26.742070 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:26.742236 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 18:33:26.742281 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 18:33:26.743399 21054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:26.737256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:41355 (local address 127.10.131.3:54679)
0620 18:33:26.738552 (+ 1296us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:26.738606 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:26.740325 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:26.740950 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:26.740957 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:26.741047 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:26.741655 (+ 608us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:33:26.741657 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:26.741692 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:33:26.741768 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:33:26.742055 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:33:26.742062 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:33:26.742068 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 18:33:26.742283 (+ 215us) client_negotiation.cc:240] Negotiation successful
0620 18:33:26.742465 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":412,"threads_started":1}
I20260620 18:33:26.744717 21052 heartbeater.cc:347] Connected to a master server at 127.10.131.62:41355
I20260620 18:33:26.745046 21052 heartbeater.cc:464] Registering TS with master...
I20260620 18:33:26.745616 21052 heartbeater.cc:511] Master 127.10.131.62:41355 requested a full tablet report, sending...
I20260620 18:33:26.746945 20605 ts_manager.cc:194] Registered new tserver with Master: 1ad3a62a30fd48a6aafde4d2da222b90 (127.10.131.3:38989)
I20260620 18:33:26.751631 10764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 18:33:26.757706 20793 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:33:26.755994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38584 (local address 127.10.131.62:41355)
0620 18:33:26.756145 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:26.756153 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:26.756632 (+ 479us) server_negotiation.cc:407] Connection header received
0620 18:33:26.756837 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:26.756886 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:26.756974 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:26.757072 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:26.757427 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38584: BlockingRecv error: recv got EOF from 127.0.0.1:38584 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 18:33:26.758414 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 20664
I20260620 18:33:26.761685 21061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:26.759381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33176 (local address 127.10.131.1:33169)
0620 18:33:26.759850 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:26.759864 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:26.759889 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:33:26.759945 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:26.759950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:26.760153 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:26.760276 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:26.761143 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:26.761150 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:26.761160 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:33:26.761171 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:26.761210 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:33:26.761226 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:26.761326 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:26.761392 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 18:33:26.761471 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":218,"threads_started":1}
I20260620 18:33:26.762777 20786 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:33:26.842075 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 20664
I20260620 18:33:26.858251 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 20795
I20260620 18:33:26.861861 21063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:26.859431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32806 (local address 127.10.131.2:46503)
0620 18:33:26.859866 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:26.859879 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:26.859904 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:33:26.859963 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:26.859968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:26.860196 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:26.860320 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:26.861206 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:26.861212 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:26.861223 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:33:26.861235 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:26.861278 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:33:26.861295 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:26.861397 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:26.861559 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 18:33:26.861644 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":181,"threads_started":1}
I20260620 18:33:26.863015 20917 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:33:26.943320 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 20795
I20260620 18:33:26.959242 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 20925
I20260620 18:33:26.963172 21065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:26.960176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54336 (local address 127.10.131.3:38989)
0620 18:33:26.960826 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:26.960833 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:26.960906 (+ 73us) server_negotiation.cc:407] Connection header received
0620 18:33:26.960970 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:26.960976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:26.961251 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:26.961420 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:26.962376 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:26.962383 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:26.962396 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:33:26.962443 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:26.962508 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:33:26.962524 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:26.962654 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:26.962735 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 18:33:26.962836 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":269,"threads_started":1}
I20260620 18:33:26.964425 21047 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:33:27.047214 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 20925
I20260620 18:33:27.065237 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 20574
I20260620 18:33:27.066385 20635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:33:27.165756 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 20574
2026-06-20T18:33:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2326 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:33:27 dist-test-slave-cbn3 krb5kdc[21070](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:33:27 dist-test-slave-cbn3 krb5kdc[21070](info): set up 2 sockets
Jun 20 18:33:27 dist-test-slave-cbn3 krb5kdc[21070](info): commencing operation
krb5kdc: starting...
W20260620 18:33:28.310086 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:33:28 dist-test-slave-cbn3 krb5kdc[21070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:33:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:33:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.131.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
I20260620 18:33:28.526444 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:40625
--webserver_interface=127.10.131.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:36521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:40625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:33:28.845039 21083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:33:28.845316 21083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:33:28.845405 21083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:33:28.852929 21083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:33:28.853065 21083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:33:28.853113 21083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:33:28.853173 21083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:33:28.853216 21083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:33:28.863091 21083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:36521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:40625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:40625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:33:28.864332 21083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:33:28.865638 21083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:33:28.877445 21088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:28.877545 21091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:28.877501 21089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:33:28.882244 21083 server_base.cc:1034] running on GCE node
I20260620 18:33:28.883078 21083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:33:28.884418 21083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:33:28.885645 21083 hybrid_clock.cc:648] HybridClock initialized: now 1781980408885592 us; error 47 us; skew 500 ppm
I20260620 18:33:28.885977 21083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:33:28 dist-test-slave-cbn3 krb5kdc[21070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980408, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:33:28.891718 21083 init.cc:377] Logged in from keytab as kudu/127.10.131.62@KRBTEST.COM (short username kudu)
I20260620 18:33:28.893229 21083 webserver.cc:460] Webserver started at http://127.10.131.62:38329/ using document root <none> and password file <none>
I20260620 18:33:28.894167 21083 fs_manager.cc:359] Metadata directory not provided
I20260620 18:33:28.894268 21083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:33:28.894523 21083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:33:28.897403 21083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "36aa6ecc73aa412d81e1559772dd35cc"
format_stamp: "Formatted at 2026-06-20 18:33:28 on dist-test-slave-cbn3"
server_key: "41ab0d27aabd62749816e5cb50ff20c1"
server_key_iv: "1a55b4d7508ae2d7dacd4b992832f0b5"
server_key_version: "encryptionkey@0"
I20260620 18:33:28.898093 21083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "36aa6ecc73aa412d81e1559772dd35cc"
format_stamp: "Formatted at 2026-06-20 18:33:28 on dist-test-slave-cbn3"
server_key: "41ab0d27aabd62749816e5cb50ff20c1"
server_key_iv: "1a55b4d7508ae2d7dacd4b992832f0b5"
server_key_version: "encryptionkey@0"
I20260620 18:33:28.902948 21083 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:33:28.906219 21098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:28.907481 21083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260620 18:33:28.907613 21083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "36aa6ecc73aa412d81e1559772dd35cc"
format_stamp: "Formatted at 2026-06-20 18:33:28 on dist-test-slave-cbn3"
server_key: "41ab0d27aabd62749816e5cb50ff20c1"
server_key_iv: "1a55b4d7508ae2d7dacd4b992832f0b5"
server_key_version: "encryptionkey@0"
I20260620 18:33:28.907768 21083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:33:28.938730 21083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:33:28.942559 21083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:33:28.942793 21083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:33:28.960909 21083 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:40625
I20260620 18:33:28.960932 21149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:40625 every 8 connection(s)
I20260620 18:33:28.962256 21083 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:33:28.965798 21150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:33:28.968418 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 21083
I20260620 18:33:28.968636 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:33:28.969458 10764 external_mini_cluster.cc:1428] Setting key 6b81270d8097485eb23ccfe17ad50aeb
I20260620 18:33:28.977423 21150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36aa6ecc73aa412d81e1559772dd35cc: Bootstrap starting.
I20260620 18:33:28.980509 21150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36aa6ecc73aa412d81e1559772dd35cc: Neither blocks nor log segments found. Creating new log.
I20260620 18:33:28.981547 21150 log.cc:824] T 00000000000000000000000000000000 P 36aa6ecc73aa412d81e1559772dd35cc: Log is configured to *not* fsync() on all Append() calls
Jun 20 18:33:28 dist-test-slave-cbn3 krb5kdc[21070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:33:28.984382 21150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36aa6ecc73aa412d81e1559772dd35cc: No bootstrap required, opened a new log
I20260620 18:33:28.991171 21153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:28.971576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55924 (local address 127.10.131.62:40625)
0620 18:33:28.972313 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:28.972325 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:28.972362 (+ 37us) server_negotiation.cc:407] Connection header received
0620 18:33:28.974028 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:28.974049 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:28.974424 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:28.974792 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:28.976186 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:28.977527 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:28.978782 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:28.979484 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:28.979678 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:28.984021 (+ 4343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:28.984028 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:28.984041 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:33:28.984079 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:28.987115 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:28.987835 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:28.987841 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:28.987847 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:28.988002 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:28.988394 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:28.988404 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:28.988409 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:28.989019 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:28.989348 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:28.989764 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 18:33:28.990182 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":325,"threads_started":1}
I20260620 18:33:28.992573 21150 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36aa6ecc73aa412d81e1559772dd35cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36aa6ecc73aa412d81e1559772dd35cc" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 40625 } }
I20260620 18:33:28.992951 21150 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36aa6ecc73aa412d81e1559772dd35cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:33:28.993045 21150 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36aa6ecc73aa412d81e1559772dd35cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36aa6ecc73aa412d81e1559772dd35cc, State: Initialized, Role: FOLLOWER
I20260620 18:33:28.993698 21150 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36aa6ecc73aa412d81e1559772dd35cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36aa6ecc73aa412d81e1559772dd35cc" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 40625 } }
I20260620 18:33:28.993969 21150 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36aa6ecc73aa412d81e1559772dd35cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:33:28.994110 21150 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36aa6ecc73aa412d81e1559772dd35cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:33:28.994256 21150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36aa6ecc73aa412d81e1559772dd35cc [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:33:28.996237 21150 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36aa6ecc73aa412d81e1559772dd35cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36aa6ecc73aa412d81e1559772dd35cc" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 40625 } }
I20260620 18:33:28.996652 21150 leader_election.cc:302] T 00000000000000000000000000000000 P 36aa6ecc73aa412d81e1559772dd35cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36aa6ecc73aa412d81e1559772dd35cc; no voters:
I20260620 18:33:28.997234 21150 leader_election.cc:288] T 00000000000000000000000000000000 P 36aa6ecc73aa412d81e1559772dd35cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:33:28.997351 21155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36aa6ecc73aa412d81e1559772dd35cc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:33:28.998030 21155 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36aa6ecc73aa412d81e1559772dd35cc [term 1 LEADER]: Becoming Leader. State: Replica: 36aa6ecc73aa412d81e1559772dd35cc, State: Running, Role: LEADER
I20260620 18:33:28.998569 21155 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36aa6ecc73aa412d81e1559772dd35cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36aa6ecc73aa412d81e1559772dd35cc" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 40625 } }
I20260620 18:33:28.999282 21150 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36aa6ecc73aa412d81e1559772dd35cc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:33:29.001711 21156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36aa6ecc73aa412d81e1559772dd35cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36aa6ecc73aa412d81e1559772dd35cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36aa6ecc73aa412d81e1559772dd35cc" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 40625 } } }
I20260620 18:33:29.001780 21157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36aa6ecc73aa412d81e1559772dd35cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36aa6ecc73aa412d81e1559772dd35cc. Latest consensus state: current_term: 1 leader_uuid: "36aa6ecc73aa412d81e1559772dd35cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36aa6ecc73aa412d81e1559772dd35cc" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 40625 } } }
I20260620 18:33:29.002187 21157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36aa6ecc73aa412d81e1559772dd35cc [sys.catalog]: This master's current role is: LEADER
I20260620 18:33:29.002079 21156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36aa6ecc73aa412d81e1559772dd35cc [sys.catalog]: This master's current role is: LEADER
I20260620 18:33:29.003882 21161 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:33:29.009661 21161 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:33:29.017779 21161 catalog_manager.cc:1347] Generated new cluster ID: 68db549e57f24d599dee7367005dd3b4
I20260620 18:33:29.017900 21161 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:33:29.028479 21161 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:33:29.030418 21161 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:33:29.042148 21161 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36aa6ecc73aa412d81e1559772dd35cc: Generated new TSK 0
I20260620 18:33:29.043056 21161 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.131.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab.
Entry for principal kudu/127.10.131.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab.
I20260620 18:33:29.099670 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.1:0
--local_ip_for_outbound_sockets=127.10.131.1
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:40625
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:36521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:33:29.425043 21176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:33:29.425328 21176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:33:29.425486 21176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:33:29.432714 21176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:33:29.432824 21176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:33:29.433027 21176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.1
I20260620 18:33:29.442744 21176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:36521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:40625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.1
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:33:29.444047 21176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:33:29.445430 21176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:33:29.458154 21181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:29.458093 21182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:29.458124 21184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:33:29.458635 21176 server_base.cc:1034] running on GCE node
I20260620 18:33:29.459257 21176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:33:29.460158 21176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:33:29.461381 21176 hybrid_clock.cc:648] HybridClock initialized: now 1781980409461325 us; error 52 us; skew 500 ppm
I20260620 18:33:29.461685 21176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:33:29 dist-test-slave-cbn3 krb5kdc[21070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980409, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:33:29.464748 21176 init.cc:377] Logged in from keytab as kudu/127.10.131.1@KRBTEST.COM (short username kudu)
I20260620 18:33:29.466261 21176 webserver.cc:460] Webserver started at http://127.10.131.1:34587/ using document root <none> and password file <none>
I20260620 18:33:29.466943 21176 fs_manager.cc:359] Metadata directory not provided
I20260620 18:33:29.467111 21176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:33:29.467433 21176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:33:29.470312 21176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/data/instance:
uuid: "f2e5540a932d455dac365f8ed9c5f699"
format_stamp: "Formatted at 2026-06-20 18:33:29 on dist-test-slave-cbn3"
server_key: "65fc1ed3efaf29c2a42fe33fa655baba"
server_key_iv: "3d66d253f254258ddc820abbbff09a38"
server_key_version: "encryptionkey@0"
I20260620 18:33:29.471029 21176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance:
uuid: "f2e5540a932d455dac365f8ed9c5f699"
format_stamp: "Formatted at 2026-06-20 18:33:29 on dist-test-slave-cbn3"
server_key: "65fc1ed3efaf29c2a42fe33fa655baba"
server_key_iv: "3d66d253f254258ddc820abbbff09a38"
server_key_version: "encryptionkey@0"
I20260620 18:33:29.475958 21176 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:33:29.479291 21191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:29.480556 21176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:33:29.480723 21176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/wal
uuid: "f2e5540a932d455dac365f8ed9c5f699"
format_stamp: "Formatted at 2026-06-20 18:33:29 on dist-test-slave-cbn3"
server_key: "65fc1ed3efaf29c2a42fe33fa655baba"
server_key_iv: "3d66d253f254258ddc820abbbff09a38"
server_key_version: "encryptionkey@0"
I20260620 18:33:29.481003 21176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:33:29.505987 21176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:33:29.510308 21176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:33:29.510614 21176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:33:29.511677 21176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:33:29.513160 21176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:33:29.513268 21176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:29.513415 21176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:33:29.513487 21176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:29.546540 21176 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.1:38675
I20260620 18:33:29.546630 21303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.1:38675 every 8 connection(s)
I20260620 18:33:29.547863 21176 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
I20260620 18:33:29.551836 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 21176
I20260620 18:33:29.552055 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance
I20260620 18:33:29.552948 10764 external_mini_cluster.cc:1428] Setting key 4fd634f9c58503e88e05c9158c7f9090
Jun 20 18:33:29 dist-test-slave-cbn3 krb5kdc[21070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980409, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:33:29.569860 21306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:29.550424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:40793 (local address 127.10.131.62:40625)
0620 18:33:29.550847 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:29.550852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:29.552615 (+ 1763us) server_negotiation.cc:407] Connection header received
0620 18:33:29.554763 (+ 2148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:29.554767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:29.554832 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:29.554951 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:29.556454 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:29.557627 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:29.559176 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:29.559907 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:29.560047 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:29.564045 (+ 3998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:29.564051 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:29.564056 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:33:29.564088 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:29.566641 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:29.567574 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:29.567578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:29.567580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:29.567691 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:29.568168 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:29.568171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:29.568173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:29.568413 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:29.568520 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:29.569444 (+ 924us) server_negotiation.cc:299] Negotiation successful
0620 18:33:29.569610 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":213,"threads_started":1}
I20260620 18:33:29.571440 21307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:29.550943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:40625 (local address 127.10.131.1:40793)
0620 18:33:29.552013 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:29.552105 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:29.554377 (+ 2272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:29.555175 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:29.555184 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:29.555693 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:29.556268 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:29.556285 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:29.557781 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:29.557785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:29.559024 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:29.559035 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:29.560108 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:29.560112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:29.560309 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:29.561056 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:29.561079 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:29.563821 (+ 2742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:33:29.566775 (+ 2954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:33:29.566787 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:33:29.566807 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:33:29.567451 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:33:29.567799 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:33:29.567806 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:33:29.567813 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:33:29.568058 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:33:29.568559 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:33:29.568571 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:33:29.569051 (+ 480us) client_negotiation.cc:769] Sending connection context
0620 18:33:29.569354 (+ 303us) client_negotiation.cc:240] Negotiation successful
0620 18:33:29.569936 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":329,"threads_started":1}
I20260620 18:33:29.573061 21304 heartbeater.cc:347] Connected to a master server at 127.10.131.62:40625
I20260620 18:33:29.573362 21304 heartbeater.cc:464] Registering TS with master...
I20260620 18:33:29.574074 21304 heartbeater.cc:511] Master 127.10.131.62:40625 requested a full tablet report, sending...
I20260620 18:33:29.576110 21115 ts_manager.cc:194] Registered new tserver with Master: f2e5540a932d455dac365f8ed9c5f699 (127.10.131.1:38675)
WARNING: no policy specified for kudu/127.10.131.2@KRBTEST.COM; defaulting to no policy
I20260620 18:33:29.577626 21115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.1@KRBTEST.COM'} at 127.10.131.1:40793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab.
Entry for principal kudu/127.10.131.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab.
I20260620 18:33:29.605901 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.2:0
--local_ip_for_outbound_sockets=127.10.131.2
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:40625
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:36521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:33:29.931216 21310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:33:29.931510 21310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:33:29.931669 21310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:33:29.938776 21310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:33:29.938928 21310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:33:29.939144 21310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.2
I20260620 18:33:29.948513 21310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:36521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:40625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.2
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:33:29.949859 21310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:33:29.951092 21310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:33:29.964160 21315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:29.964196 21316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:29.964229 21318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:33:29.965955 21310 server_base.cc:1034] running on GCE node
I20260620 18:33:29.966658 21310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:33:29.967689 21310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:33:29.968928 21310 hybrid_clock.cc:648] HybridClock initialized: now 1781980409968867 us; error 52 us; skew 500 ppm
I20260620 18:33:29.969290 21310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:33:29 dist-test-slave-cbn3 krb5kdc[21070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980409, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:33:29.972725 21310 init.cc:377] Logged in from keytab as kudu/127.10.131.2@KRBTEST.COM (short username kudu)
I20260620 18:33:29.978896 21310 webserver.cc:460] Webserver started at http://127.10.131.2:43497/ using document root <none> and password file <none>
I20260620 18:33:29.979573 21310 fs_manager.cc:359] Metadata directory not provided
I20260620 18:33:29.979732 21310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:33:29.979985 21310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:33:29.982761 21310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/data/instance:
uuid: "a75d97b23ea04dd485e5f597fc156bdd"
format_stamp: "Formatted at 2026-06-20 18:33:29 on dist-test-slave-cbn3"
server_key: "9e5d0f949445a5ea9c6f34646d0e34bf"
server_key_iv: "63d023ab12e56b9910611db4802e7f52"
server_key_version: "encryptionkey@0"
I20260620 18:33:29.983428 21310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance:
uuid: "a75d97b23ea04dd485e5f597fc156bdd"
format_stamp: "Formatted at 2026-06-20 18:33:29 on dist-test-slave-cbn3"
server_key: "9e5d0f949445a5ea9c6f34646d0e34bf"
server_key_iv: "63d023ab12e56b9910611db4802e7f52"
server_key_version: "encryptionkey@0"
I20260620 18:33:29.988513 21310 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 18:33:29.991901 21325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:29.993307 21310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:33:29.993427 21310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/wal
uuid: "a75d97b23ea04dd485e5f597fc156bdd"
format_stamp: "Formatted at 2026-06-20 18:33:29 on dist-test-slave-cbn3"
server_key: "9e5d0f949445a5ea9c6f34646d0e34bf"
server_key_iv: "63d023ab12e56b9910611db4802e7f52"
server_key_version: "encryptionkey@0"
I20260620 18:33:29.993574 21310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:33:30.041461 21310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:33:30.046033 21310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:33:30.046307 21310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:33:30.047223 21310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:33:30.048569 21310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:33:30.048641 21310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:30.048727 21310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:33:30.048769 21310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:30.081934 21310 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.2:39379
I20260620 18:33:30.082038 21437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.2:39379 every 8 connection(s)
I20260620 18:33:30.083201 21310 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
I20260620 18:33:30.091871 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 21310
I20260620 18:33:30.092098 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance
I20260620 18:33:30.092514 10764 external_mini_cluster.cc:1428] Setting key b47725bebe6f8fc0b6451e4e47241e95
Jun 20 18:33:30 dist-test-slave-cbn3 krb5kdc[21070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980409, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:33:30.104490 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:30.085686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:34389 (local address 127.10.131.62:40625)
0620 18:33:30.086128 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:30.086132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:30.086865 (+ 733us) server_negotiation.cc:407] Connection header received
0620 18:33:30.088618 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:30.088624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:30.088717 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:30.088834 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:30.090327 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:30.091055 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:30.092774 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:30.093292 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:30.093413 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:30.097126 (+ 3713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:30.097155 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:30.097164 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:33:30.097212 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:30.101089 (+ 3877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:30.101808 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:30.101816 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:30.101822 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:30.102010 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:30.102469 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:30.102477 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:30.102483 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:30.102898 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:30.103071 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:30.103813 (+ 742us) server_negotiation.cc:299] Negotiation successful
0620 18:33:30.104038 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":260,"threads_started":1}
I20260620 18:33:30.106005 21441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:30.086091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:40625 (local address 127.10.131.2:34389)
0620 18:33:30.086722 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:30.086758 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:30.088362 (+ 1604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:30.089097 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:30.089105 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:30.089610 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:30.090157 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:30.090173 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:30.091206 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:30.091228 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:30.092639 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:30.092652 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:30.093531 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:30.093538 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:30.093818 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:30.094668 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:30.094704 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:30.096877 (+ 2173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:33:30.101231 (+ 4354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:33:30.101238 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:33:30.101251 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:33:30.101681 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:33:30.102130 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:33:30.102133 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:33:30.102135 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:33:30.102307 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:33:30.103047 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:33:30.103056 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:33:30.103407 (+ 351us) client_negotiation.cc:769] Sending connection context
0620 18:33:30.103742 (+ 335us) client_negotiation.cc:240] Negotiation successful
0620 18:33:30.104305 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":250,"threads_started":1}
I20260620 18:33:30.107479 21438 heartbeater.cc:347] Connected to a master server at 127.10.131.62:40625
I20260620 18:33:30.107786 21438 heartbeater.cc:464] Registering TS with master...
I20260620 18:33:30.108641 21438 heartbeater.cc:511] Master 127.10.131.62:40625 requested a full tablet report, sending...
I20260620 18:33:30.110163 21115 ts_manager.cc:194] Registered new tserver with Master: a75d97b23ea04dd485e5f597fc156bdd (127.10.131.2:39379)
I20260620 18:33:30.111053 21115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.2@KRBTEST.COM'} at 127.10.131.2:34389
WARNING: no policy specified for kudu/127.10.131.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab.
Entry for principal kudu/127.10.131.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab.
I20260620 18:33:30.147485 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.3:0
--local_ip_for_outbound_sockets=127.10.131.3
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:40625
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:36521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:33:30.462284 21444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:33:30.462538 21444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:33:30.462625 21444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:33:30.469686 21444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:33:30.469779 21444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:33:30.469880 21444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.3
I20260620 18:33:30.479049 21444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:36521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:40625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.3
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:33:30.480219 21444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:33:30.481389 21444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:33:30.498128 21449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:30.498081 21450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:33:30.498107 21452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:33:30.498857 21444 server_base.cc:1034] running on GCE node
I20260620 18:33:30.499430 21444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:33:30.500305 21444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:33:30.501523 21444 hybrid_clock.cc:648] HybridClock initialized: now 1781980410501482 us; error 55 us; skew 500 ppm
I20260620 18:33:30.501864 21444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:33:30 dist-test-slave-cbn3 krb5kdc[21070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980410, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:33:30.505028 21444 init.cc:377] Logged in from keytab as kudu/127.10.131.3@KRBTEST.COM (short username kudu)
I20260620 18:33:30.506884 21444 webserver.cc:460] Webserver started at http://127.10.131.3:36577/ using document root <none> and password file <none>
I20260620 18:33:30.507507 21444 fs_manager.cc:359] Metadata directory not provided
I20260620 18:33:30.507601 21444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:33:30.507841 21444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:33:30.510813 21444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/data/instance:
uuid: "4052766d900c481887e1b157069b8fbc"
format_stamp: "Formatted at 2026-06-20 18:33:30 on dist-test-slave-cbn3"
server_key: "40b6ef9f325247727bcdca68ba568c58"
server_key_iv: "6dc4d802a0414796307945a4d32c1923"
server_key_version: "encryptionkey@0"
I20260620 18:33:30.511510 21444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance:
uuid: "4052766d900c481887e1b157069b8fbc"
format_stamp: "Formatted at 2026-06-20 18:33:30 on dist-test-slave-cbn3"
server_key: "40b6ef9f325247727bcdca68ba568c58"
server_key_iv: "6dc4d802a0414796307945a4d32c1923"
server_key_version: "encryptionkey@0"
I20260620 18:33:30.516484 21444 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:33:30.519845 21459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:30.521243 21444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 18:33:30.521369 21444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/wal
uuid: "4052766d900c481887e1b157069b8fbc"
format_stamp: "Formatted at 2026-06-20 18:33:30 on dist-test-slave-cbn3"
server_key: "40b6ef9f325247727bcdca68ba568c58"
server_key_iv: "6dc4d802a0414796307945a4d32c1923"
server_key_version: "encryptionkey@0"
I20260620 18:33:30.521533 21444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:33:30.533179 21444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:33:30.536758 21444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:33:30.537061 21444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:33:30.537917 21444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:33:30.539114 21444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:33:30.539193 21444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:30.539273 21444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:33:30.539328 21444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:33:30.572119 21444 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.3:37711
I20260620 18:33:30.572201 21571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.3:37711 every 8 connection(s)
I20260620 18:33:30.573438 21444 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
I20260620 18:33:30.578536 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 21444
I20260620 18:33:30.578742 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance
I20260620 18:33:30.579211 10764 external_mini_cluster.cc:1428] Setting key 6a9cc5b518786d5851e7e042907ca672
I20260620 18:33:30.580925 21304 heartbeater.cc:503] Master 127.10.131.62:40625 was elected leader, sending a full tablet report...
Jun 20 18:33:30 dist-test-slave-cbn3 krb5kdc[21070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980410, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:33:30.594374 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:30.575989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:43569 (local address 127.10.131.62:40625)
0620 18:33:30.576168 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:30.576173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:30.577273 (+ 1100us) server_negotiation.cc:407] Connection header received
0620 18:33:30.579039 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:30.579045 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:30.579137 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:30.579252 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:30.580966 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:30.582123 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:30.584404 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:30.585136 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:30.585292 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:30.588714 (+ 3422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:30.588718 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:30.588721 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:33:30.588751 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:30.591427 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:30.592071 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:30.592074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:30.592076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:30.592185 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:30.592554 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:30.592558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:30.592561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:30.592981 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:30.593135 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:30.593831 (+ 696us) server_negotiation.cc:299] Negotiation successful
0620 18:33:30.594062 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:33:30.595383 21574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:30.576398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:40625 (local address 127.10.131.3:43569)
0620 18:33:30.577110 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:30.577146 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:30.578760 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:30.579475 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:30.579483 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:30.580072 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:33:30.580741 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:30.580767 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:30.582322 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:30.582328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:30.584162 (+ 1834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:30.584200 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:30.585368 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:30.585375 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:30.585673 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:30.586466 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:33:30.586487 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:33:30.588510 (+ 2023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:33:30.591551 (+ 3041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:33:30.591558 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:33:30.591569 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:33:30.591973 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:33:30.592273 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:33:30.592276 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:33:30.592279 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:33:30.592441 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:33:30.593115 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:33:30.593122 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:33:30.593407 (+ 285us) client_negotiation.cc:769] Sending connection context
0620 18:33:30.593715 (+ 308us) client_negotiation.cc:240] Negotiation successful
0620 18:33:30.594212 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":256,"threads_started":1}
I20260620 18:33:30.596679 21572 heartbeater.cc:347] Connected to a master server at 127.10.131.62:40625
I20260620 18:33:30.597062 21572 heartbeater.cc:464] Registering TS with master...
I20260620 18:33:30.597815 21572 heartbeater.cc:511] Master 127.10.131.62:40625 requested a full tablet report, sending...
I20260620 18:33:30.599248 21114 ts_manager.cc:194] Registered new tserver with Master: 4052766d900c481887e1b157069b8fbc (127.10.131.3:37711)
I20260620 18:33:30.600564 21114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.3@KRBTEST.COM'} at 127.10.131.3:43569
I20260620 18:33:30.606841 10764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:33:30.622859 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:30.611146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55932 (local address 127.10.131.62:40625)
0620 18:33:30.611306 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:30.611311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:30.611801 (+ 490us) server_negotiation.cc:407] Connection header received
0620 18:33:30.611978 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:30.611982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:30.612058 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:30.612137 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:30.613543 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:30.614246 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:30.615392 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:30.615800 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:30.615901 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:30.618052 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:30.618056 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:30.618059 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:33:30.618088 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:30.620556 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:30.621184 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:30.621188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:30.621190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:30.621307 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:30.621630 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:30.621633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:30.621634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:30.621925 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:30.622102 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:30.622353 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 18:33:30.622501 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 18:33:30.626961 21114 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:33:30.633709 21114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:33:30.667152 21586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:30.645337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:38675 (local address 127.0.0.1:50906)
0620 18:33:30.646829 (+ 1492us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:30.646872 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:30.647030 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:30.659149 (+ 12119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:30.659159 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:30.659205 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:33:30.660065 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:30.660076 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:30.662138 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:30.662143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:30.664591 (+ 2448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:30.664603 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:30.666082 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:30.666088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:30.666288 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:30.666343 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 18:33:30.666472 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 18:33:30.666684 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1274,"mutex_wait_us":122,"spinlock_wait_cycles":4992,"thread_start_us":2027,"threads_started":1}
I20260620 18:33:30.667425 21587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:30.646065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53612 (local address 127.10.131.2:39379)
0620 18:33:30.658283 (+ 12218us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:30.658294 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:30.658331 (+ 37us) server_negotiation.cc:407] Connection header received
0620 18:33:30.658459 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:30.658470 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:30.658809 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:30.659092 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:33:30.660269 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:30.661984 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:30.664962 (+ 2978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:30.666309 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:30.666464 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:30.666643 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:30.666915 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 18:33:30.667081 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12009,"thread_start_us":11865,"threads_started":1}
I20260620 18:33:30.668066 21585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:30.645654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50906 (local address 127.10.131.1:38675)
0620 18:33:30.658635 (+ 12981us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:30.658642 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:30.658665 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:33:30.658741 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:30.658746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:30.658899 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:30.659756 (+ 857us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:33:30.660469 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:30.662009 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:30.664754 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:30.665879 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:30.666016 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:30.666281 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:30.667593 (+ 1312us) server_negotiation.cc:299] Negotiation successful
0620 18:33:30.667782 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12832,"thread_start_us":12693,"threads_started":1}
I20260620 18:33:30.668197 21584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:30.645915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:39379 (local address 127.0.0.1:53612)
0620 18:33:30.647604 (+ 1689us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:30.647620 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:30.647724 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:30.659149 (+ 11425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:30.659159 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:30.659197 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:33:30.660033 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:30.660055 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:30.662166 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:30.662170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:30.664728 (+ 2558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:30.664743 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:30.666525 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:30.666531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:30.666710 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:30.666745 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 18:33:30.666840 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 18:33:30.666969 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1570,"mutex_wait_us":216,"thread_start_us":574,"threads_started":1}
I20260620 18:33:30.669517 21588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:30.647895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:37711 (local address 127.0.0.1:56188)
0620 18:33:30.659966 (+ 12071us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:30.659997 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:30.660490 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:30.661116 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:30.661121 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:30.661145 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:33:30.661478 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:30.661487 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:30.663209 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:30.663214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:30.665545 (+ 2331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:30.665559 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:30.668637 (+ 3078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:30.668647 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:30.668876 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:30.668945 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 18:33:30.669042 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 18:33:30.669163 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11904,"mutex_wait_us":1,"thread_start_us":11733,"threads_started":1}
I20260620 18:33:30.670110 21589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:30.648305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56188 (local address 127.10.131.3:37711)
0620 18:33:30.657937 (+ 9632us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:30.657949 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:30.660192 (+ 2243us) server_negotiation.cc:407] Connection header received
0620 18:33:30.660691 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:30.660699 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:30.660945 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:30.661124 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:33:30.661638 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:30.663052 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:30.665727 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:30.667158 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:30.667939 (+ 781us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:30.668119 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:30.669083 (+ 964us) server_negotiation.cc:299] Negotiation successful
0620 18:33:30.669255 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9450,"thread_start_us":9286,"threads_started":1}
I20260620 18:33:30.671788 21373 tablet_service.cc:1467] Processing CreateTablet for tablet 786611b0db5b4e1ea3b496a930b2a177 (DEFAULT_TABLE table=test-table [id=58eeade460cc4a39b5ddd57daecae08c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:33:30.671828 21239 tablet_service.cc:1467] Processing CreateTablet for tablet 786611b0db5b4e1ea3b496a930b2a177 (DEFAULT_TABLE table=test-table [id=58eeade460cc4a39b5ddd57daecae08c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:33:30.673211 21239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 786611b0db5b4e1ea3b496a930b2a177. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:33:30.673226 21373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 786611b0db5b4e1ea3b496a930b2a177. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:33:30.673498 21507 tablet_service.cc:1467] Processing CreateTablet for tablet 786611b0db5b4e1ea3b496a930b2a177 (DEFAULT_TABLE table=test-table [id=58eeade460cc4a39b5ddd57daecae08c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:33:30.674721 21507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 786611b0db5b4e1ea3b496a930b2a177. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:33:30.684051 21590 tablet_bootstrap.cc:492] T 786611b0db5b4e1ea3b496a930b2a177 P a75d97b23ea04dd485e5f597fc156bdd: Bootstrap starting.
I20260620 18:33:30.687585 21590 tablet_bootstrap.cc:654] T 786611b0db5b4e1ea3b496a930b2a177 P a75d97b23ea04dd485e5f597fc156bdd: Neither blocks nor log segments found. Creating new log.
I20260620 18:33:30.688632 21592 tablet_bootstrap.cc:492] T 786611b0db5b4e1ea3b496a930b2a177 P 4052766d900c481887e1b157069b8fbc: Bootstrap starting.
I20260620 18:33:30.688956 21590 log.cc:824] T 786611b0db5b4e1ea3b496a930b2a177 P a75d97b23ea04dd485e5f597fc156bdd: Log is configured to *not* fsync() on all Append() calls
I20260620 18:33:30.692118 21591 tablet_bootstrap.cc:492] T 786611b0db5b4e1ea3b496a930b2a177 P f2e5540a932d455dac365f8ed9c5f699: Bootstrap starting.
I20260620 18:33:30.692732 21592 tablet_bootstrap.cc:654] T 786611b0db5b4e1ea3b496a930b2a177 P 4052766d900c481887e1b157069b8fbc: Neither blocks nor log segments found. Creating new log.
I20260620 18:33:30.693418 21590 tablet_bootstrap.cc:492] T 786611b0db5b4e1ea3b496a930b2a177 P a75d97b23ea04dd485e5f597fc156bdd: No bootstrap required, opened a new log
I20260620 18:33:30.693704 21590 ts_tablet_manager.cc:1397] T 786611b0db5b4e1ea3b496a930b2a177 P a75d97b23ea04dd485e5f597fc156bdd: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.001s
I20260620 18:33:30.693974 21592 log.cc:824] T 786611b0db5b4e1ea3b496a930b2a177 P 4052766d900c481887e1b157069b8fbc: Log is configured to *not* fsync() on all Append() calls
I20260620 18:33:30.695256 21591 tablet_bootstrap.cc:654] T 786611b0db5b4e1ea3b496a930b2a177 P f2e5540a932d455dac365f8ed9c5f699: Neither blocks nor log segments found. Creating new log.
I20260620 18:33:30.696305 21591 log.cc:824] T 786611b0db5b4e1ea3b496a930b2a177 P f2e5540a932d455dac365f8ed9c5f699: Log is configured to *not* fsync() on all Append() calls
I20260620 18:33:30.699103 21591 tablet_bootstrap.cc:492] T 786611b0db5b4e1ea3b496a930b2a177 P f2e5540a932d455dac365f8ed9c5f699: No bootstrap required, opened a new log
I20260620 18:33:30.699358 21591 ts_tablet_manager.cc:1397] T 786611b0db5b4e1ea3b496a930b2a177 P f2e5540a932d455dac365f8ed9c5f699: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260620 18:33:30.703042 21590 raft_consensus.cc:348] T 786611b0db5b4e1ea3b496a930b2a177 P a75d97b23ea04dd485e5f597fc156bdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e5540a932d455dac365f8ed9c5f699" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38675 } } peers { permanent_uuid: "a75d97b23ea04dd485e5f597fc156bdd" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39379 } } peers { permanent_uuid: "4052766d900c481887e1b157069b8fbc" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37711 } }
I20260620 18:33:30.703490 21590 raft_consensus.cc:374] T 786611b0db5b4e1ea3b496a930b2a177 P a75d97b23ea04dd485e5f597fc156bdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:33:30.703591 21590 raft_consensus.cc:729] T 786611b0db5b4e1ea3b496a930b2a177 P a75d97b23ea04dd485e5f597fc156bdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a75d97b23ea04dd485e5f597fc156bdd, State: Initialized, Role: FOLLOWER
I20260620 18:33:30.704205 21592 tablet_bootstrap.cc:492] T 786611b0db5b4e1ea3b496a930b2a177 P 4052766d900c481887e1b157069b8fbc: No bootstrap required, opened a new log
I20260620 18:33:30.704231 21590 consensus_queue.cc:260] T 786611b0db5b4e1ea3b496a930b2a177 P a75d97b23ea04dd485e5f597fc156bdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e5540a932d455dac365f8ed9c5f699" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38675 } } peers { permanent_uuid: "a75d97b23ea04dd485e5f597fc156bdd" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39379 } } peers { permanent_uuid: "4052766d900c481887e1b157069b8fbc" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37711 } }
I20260620 18:33:30.704473 21592 ts_tablet_manager.cc:1397] T 786611b0db5b4e1ea3b496a930b2a177 P 4052766d900c481887e1b157069b8fbc: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.003s
I20260620 18:33:30.706079 21438 heartbeater.cc:503] Master 127.10.131.62:40625 was elected leader, sending a full tablet report...
I20260620 18:33:30.706903 21590 ts_tablet_manager.cc:1428] T 786611b0db5b4e1ea3b496a930b2a177 P a75d97b23ea04dd485e5f597fc156bdd: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260620 18:33:30.709353 21591 raft_consensus.cc:348] T 786611b0db5b4e1ea3b496a930b2a177 P f2e5540a932d455dac365f8ed9c5f699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e5540a932d455dac365f8ed9c5f699" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38675 } } peers { permanent_uuid: "a75d97b23ea04dd485e5f597fc156bdd" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39379 } } peers { permanent_uuid: "4052766d900c481887e1b157069b8fbc" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37711 } }
I20260620 18:33:30.709800 21591 raft_consensus.cc:374] T 786611b0db5b4e1ea3b496a930b2a177 P f2e5540a932d455dac365f8ed9c5f699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:33:30.709909 21591 raft_consensus.cc:729] T 786611b0db5b4e1ea3b496a930b2a177 P f2e5540a932d455dac365f8ed9c5f699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2e5540a932d455dac365f8ed9c5f699, State: Initialized, Role: FOLLOWER
I20260620 18:33:30.710647 21591 consensus_queue.cc:260] T 786611b0db5b4e1ea3b496a930b2a177 P f2e5540a932d455dac365f8ed9c5f699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e5540a932d455dac365f8ed9c5f699" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38675 } } peers { permanent_uuid: "a75d97b23ea04dd485e5f597fc156bdd" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39379 } } peers { permanent_uuid: "4052766d900c481887e1b157069b8fbc" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37711 } }
I20260620 18:33:30.712442 21591 ts_tablet_manager.cc:1428] T 786611b0db5b4e1ea3b496a930b2a177 P f2e5540a932d455dac365f8ed9c5f699: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260620 18:33:30.714840 21592 raft_consensus.cc:348] T 786611b0db5b4e1ea3b496a930b2a177 P 4052766d900c481887e1b157069b8fbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e5540a932d455dac365f8ed9c5f699" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38675 } } peers { permanent_uuid: "a75d97b23ea04dd485e5f597fc156bdd" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39379 } } peers { permanent_uuid: "4052766d900c481887e1b157069b8fbc" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37711 } }
I20260620 18:33:30.715303 21592 raft_consensus.cc:374] T 786611b0db5b4e1ea3b496a930b2a177 P 4052766d900c481887e1b157069b8fbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:33:30.715415 21592 raft_consensus.cc:729] T 786611b0db5b4e1ea3b496a930b2a177 P 4052766d900c481887e1b157069b8fbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4052766d900c481887e1b157069b8fbc, State: Initialized, Role: FOLLOWER
I20260620 18:33:30.715991 21592 consensus_queue.cc:260] T 786611b0db5b4e1ea3b496a930b2a177 P 4052766d900c481887e1b157069b8fbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e5540a932d455dac365f8ed9c5f699" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38675 } } peers { permanent_uuid: "a75d97b23ea04dd485e5f597fc156bdd" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39379 } } peers { permanent_uuid: "4052766d900c481887e1b157069b8fbc" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37711 } }
I20260620 18:33:30.716946 21572 heartbeater.cc:503] Master 127.10.131.62:40625 was elected leader, sending a full tablet report...
I20260620 18:33:30.717509 21592 ts_tablet_manager.cc:1428] T 786611b0db5b4e1ea3b496a930b2a177 P 4052766d900c481887e1b157069b8fbc: Time spent starting tablet: real 0.013s user 0.005s sys 0.008s
W20260620 18:33:30.800345 21305 tablet.cc:2369] T 786611b0db5b4e1ea3b496a930b2a177 P f2e5540a932d455dac365f8ed9c5f699: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:33:30.825116 21573 tablet.cc:2369] T 786611b0db5b4e1ea3b496a930b2a177 P 4052766d900c481887e1b157069b8fbc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:33:30.830869 21598 raft_consensus.cc:482] T 786611b0db5b4e1ea3b496a930b2a177 P 4052766d900c481887e1b157069b8fbc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:33:30.831200 21598 raft_consensus.cc:504] T 786611b0db5b4e1ea3b496a930b2a177 P 4052766d900c481887e1b157069b8fbc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e5540a932d455dac365f8ed9c5f699" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38675 } } peers { permanent_uuid: "a75d97b23ea04dd485e5f597fc156bdd" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39379 } } peers { permanent_uuid: "4052766d900c481887e1b157069b8fbc" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37711 } }
I20260620 18:33:30.832960 21598 leader_election.cc:288] T 786611b0db5b4e1ea3b496a930b2a177 P 4052766d900c481887e1b157069b8fbc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2e5540a932d455dac365f8ed9c5f699 (127.10.131.1:38675), a75d97b23ea04dd485e5f597fc156bdd (127.10.131.2:39379)
W20260620 18:33:30.835259 21439 tablet.cc:2369] T 786611b0db5b4e1ea3b496a930b2a177 P a75d97b23ea04dd485e5f597fc156bdd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:33:30.839111 21585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:30.833293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:58255 (local address 127.10.131.1:38675)
0620 18:33:30.833511 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:30.833520 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:30.833550 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:33:30.833709 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:30.833717 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:30.833815 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:30.833918 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:33:30.834384 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:30.835554 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:30.837409 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:30.838208 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:30.838392 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:30.838466 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:30.838750 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 18:33:30.838873 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:33:30.839134 21574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:30.833199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:38675 (local address 127.10.131.3:58255)
0620 18:33:30.833371 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:30.833392 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:30.833512 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:30.833962 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:30.833968 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:30.834009 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:33:30.834210 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:30.834226 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:30.835715 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:30.835719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:30.837257 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:30.837269 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:30.838372 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:30.838379 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:30.838562 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:30.838586 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 18:33:30.838716 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 18:33:30.838843 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":29}
I20260620 18:33:30.840011 21587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:30.833580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:37561 (local address 127.10.131.2:39379)
0620 18:33:30.833801 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:30.833808 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:30.834297 (+ 489us) server_negotiation.cc:407] Connection header received
0620 18:33:30.834455 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:30.834461 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:30.834535 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:30.834626 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:33:30.835142 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:30.836077 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:30.838138 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:30.839002 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:30.839201 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:30.839283 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:30.839621 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 18:33:30.839760 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:33:30.840047 21599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:30.833277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:39379 (local address 127.10.131.3:37561)
0620 18:33:30.834172 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0620 18:33:30.834189 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:33:30.834311 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:33:30.834641 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:33:30.834643 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:33:30.834661 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:33:30.834904 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:30.834910 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:30.836343 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:30.836348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:30.837884 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:33:30.837897 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:33:30.839234 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:33:30.839241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:33:30.839426 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:30.839473 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 18:33:30.839582 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 18:33:30.839719 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"mutex_wait_us":4,"thread_start_us":501,"threads_started":1}
I20260620 18:33:30.840230 21259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786611b0db5b4e1ea3b496a930b2a177" candidate_uuid: "4052766d900c481887e1b157069b8fbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2e5540a932d455dac365f8ed9c5f699" is_pre_election: true
I20260620 18:33:30.840867 21259 raft_consensus.cc:2454] T 786611b0db5b4e1ea3b496a930b2a177 P f2e5540a932d455dac365f8ed9c5f699 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4052766d900c481887e1b157069b8fbc in term 0.
I20260620 18:33:30.841091 21393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786611b0db5b4e1ea3b496a930b2a177" candidate_uuid: "4052766d900c481887e1b157069b8fbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a75d97b23ea04dd485e5f597fc156bdd" is_pre_election: true
I20260620 18:33:30.841569 21393 raft_consensus.cc:2454] T 786611b0db5b4e1ea3b496a930b2a177 P a75d97b23ea04dd485e5f597fc156bdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4052766d900c481887e1b157069b8fbc in term 0.
I20260620 18:33:30.841609 21460 leader_election.cc:302] T 786611b0db5b4e1ea3b496a930b2a177 P 4052766d900c481887e1b157069b8fbc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4052766d900c481887e1b157069b8fbc, f2e5540a932d455dac365f8ed9c5f699; no voters:
I20260620 18:33:30.842046 21598 raft_consensus.cc:2781] T 786611b0db5b4e1ea3b496a930b2a177 P 4052766d900c481887e1b157069b8fbc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:33:30.842186 21598 raft_consensus.cc:482] T 786611b0db5b4e1ea3b496a930b2a177 P 4052766d900c481887e1b157069b8fbc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:33:30.842310 21598 raft_consensus.cc:3037] T 786611b0db5b4e1ea3b496a930b2a177 P 4052766d900c481887e1b157069b8fbc [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:33:30.844436 21598 raft_consensus.cc:504] T 786611b0db5b4e1ea3b496a930b2a177 P 4052766d900c481887e1b157069b8fbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e5540a932d455dac365f8ed9c5f699" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38675 } } peers { permanent_uuid: "a75d97b23ea04dd485e5f597fc156bdd" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39379 } } peers { permanent_uuid: "4052766d900c481887e1b157069b8fbc" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37711 } }
I20260620 18:33:30.845237 21598 leader_election.cc:288] T 786611b0db5b4e1ea3b496a930b2a177 P 4052766d900c481887e1b157069b8fbc [CANDIDATE]: Term 1 election: Requested vote from peers f2e5540a932d455dac365f8ed9c5f699 (127.10.131.1:38675), a75d97b23ea04dd485e5f597fc156bdd (127.10.131.2:39379)
I20260620 18:33:30.845460 21259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786611b0db5b4e1ea3b496a930b2a177" candidate_uuid: "4052766d900c481887e1b157069b8fbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2e5540a932d455dac365f8ed9c5f699"
I20260620 18:33:30.845525 21393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786611b0db5b4e1ea3b496a930b2a177" candidate_uuid: "4052766d900c481887e1b157069b8fbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a75d97b23ea04dd485e5f597fc156bdd"
I20260620 18:33:30.845674 21259 raft_consensus.cc:3037] T 786611b0db5b4e1ea3b496a930b2a177 P f2e5540a932d455dac365f8ed9c5f699 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:33:30.845696 21393 raft_consensus.cc:3037] T 786611b0db5b4e1ea3b496a930b2a177 P a75d97b23ea04dd485e5f597fc156bdd [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:33:30.848793 21393 raft_consensus.cc:2454] T 786611b0db5b4e1ea3b496a930b2a177 P a75d97b23ea04dd485e5f597fc156bdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4052766d900c481887e1b157069b8fbc in term 1.
I20260620 18:33:30.848824 21259 raft_consensus.cc:2454] T 786611b0db5b4e1ea3b496a930b2a177 P f2e5540a932d455dac365f8ed9c5f699 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4052766d900c481887e1b157069b8fbc in term 1.
I20260620 18:33:30.849223 21460 leader_election.cc:302] T 786611b0db5b4e1ea3b496a930b2a177 P 4052766d900c481887e1b157069b8fbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4052766d900c481887e1b157069b8fbc, a75d97b23ea04dd485e5f597fc156bdd, f2e5540a932d455dac365f8ed9c5f699; no voters:
I20260620 18:33:30.849530 21598 raft_consensus.cc:2781] T 786611b0db5b4e1ea3b496a930b2a177 P 4052766d900c481887e1b157069b8fbc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:33:30.849951 21598 raft_consensus.cc:686] T 786611b0db5b4e1ea3b496a930b2a177 P 4052766d900c481887e1b157069b8fbc [term 1 LEADER]: Becoming Leader. State: Replica: 4052766d900c481887e1b157069b8fbc, State: Running, Role: LEADER
I20260620 18:33:30.850407 21598 consensus_queue.cc:237] T 786611b0db5b4e1ea3b496a930b2a177 P 4052766d900c481887e1b157069b8fbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e5540a932d455dac365f8ed9c5f699" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38675 } } peers { permanent_uuid: "a75d97b23ea04dd485e5f597fc156bdd" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39379 } } peers { permanent_uuid: "4052766d900c481887e1b157069b8fbc" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37711 } }
I20260620 18:33:30.854991 21114 catalog_manager.cc:5495] T 786611b0db5b4e1ea3b496a930b2a177 P 4052766d900c481887e1b157069b8fbc reported cstate change: term changed from 0 to 1, leader changed from <none> to 4052766d900c481887e1b157069b8fbc (127.10.131.3). New cstate: current_term: 1 leader_uuid: "4052766d900c481887e1b157069b8fbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e5540a932d455dac365f8ed9c5f699" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a75d97b23ea04dd485e5f597fc156bdd" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4052766d900c481887e1b157069b8fbc" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37711 } health_report { overall_health: HEALTHY } } }
I20260620 18:33:30.903854 21589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:30.898271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56204 (local address 127.10.131.3:37711)
0620 18:33:30.898469 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:30.898476 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:30.898575 (+ 99us) server_negotiation.cc:407] Connection header received
0620 18:33:30.898820 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:30.898823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:30.898895 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:30.899046 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:33:30.899439 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:30.900250 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:30.901719 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:30.902118 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:30.902278 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:30.902762 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:33:30.903261 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:33:30.903420 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:30.903498 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 18:33:30.903577 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:33:30.912163 21393 raft_consensus.cc:1261] T 786611b0db5b4e1ea3b496a930b2a177 P a75d97b23ea04dd485e5f597fc156bdd [term 1 FOLLOWER]: Refusing update from remote peer 4052766d900c481887e1b157069b8fbc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:33:30.912184 21259 raft_consensus.cc:1261] T 786611b0db5b4e1ea3b496a930b2a177 P f2e5540a932d455dac365f8ed9c5f699 [term 1 FOLLOWER]: Refusing update from remote peer 4052766d900c481887e1b157069b8fbc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:33:30.913158 21600 consensus_queue.cc:1035] T 786611b0db5b4e1ea3b496a930b2a177 P 4052766d900c481887e1b157069b8fbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2e5540a932d455dac365f8ed9c5f699" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:33:30.913506 21598 consensus_queue.cc:1035] T 786611b0db5b4e1ea3b496a930b2a177 P 4052766d900c481887e1b157069b8fbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a75d97b23ea04dd485e5f597fc156bdd" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:33:30.925735 21604 mvcc.cc:205] Tried to move back new op lower bound from 7298991763085168640 to 7298991762846576640. Current Snapshot: MvccSnapshot[applied={T|T < 7298991763085168640}]
I20260620 18:33:30.938593 21114 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:33:30.938843 21114 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:33:30.942054 21114 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36aa6ecc73aa412d81e1559772dd35cc: Sending DeleteTablet for 3 replicas of tablet 786611b0db5b4e1ea3b496a930b2a177
I20260620 18:33:30.943158 21239 tablet_service.cc:1514] Processing DeleteTablet for tablet 786611b0db5b4e1ea3b496a930b2a177 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:33:30 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:50906
I20260620 18:33:30.943225 21373 tablet_service.cc:1514] Processing DeleteTablet for tablet 786611b0db5b4e1ea3b496a930b2a177 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:33:30 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:53612
I20260620 18:33:30.943471 21507 tablet_service.cc:1514] Processing DeleteTablet for tablet 786611b0db5b4e1ea3b496a930b2a177 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:33:30 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:56188
I20260620 18:33:30.944591 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 21176
I20260620 18:33:30.946798 21615 tablet_replica.cc:330] T 786611b0db5b4e1ea3b496a930b2a177 P 4052766d900c481887e1b157069b8fbc: stopping tablet replica
I20260620 18:33:30.947280 21615 raft_consensus.cc:2229] T 786611b0db5b4e1ea3b496a930b2a177 P 4052766d900c481887e1b157069b8fbc [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:33:30.948027 21615 raft_consensus.cc:2258] T 786611b0db5b4e1ea3b496a930b2a177 P 4052766d900c481887e1b157069b8fbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:33:30.955703 21613 tablet_replica.cc:330] T 786611b0db5b4e1ea3b496a930b2a177 P f2e5540a932d455dac365f8ed9c5f699: stopping tablet replica
I20260620 18:33:30.955859 21615 ts_tablet_manager.cc:1905] T 786611b0db5b4e1ea3b496a930b2a177 P 4052766d900c481887e1b157069b8fbc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:33:30.956166 21613 raft_consensus.cc:2229] T 786611b0db5b4e1ea3b496a930b2a177 P f2e5540a932d455dac365f8ed9c5f699 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:33:30.956683 21613 raft_consensus.cc:2258] T 786611b0db5b4e1ea3b496a930b2a177 P f2e5540a932d455dac365f8ed9c5f699 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:33:30.958817 21613 ts_tablet_manager.cc:1905] T 786611b0db5b4e1ea3b496a930b2a177 P f2e5540a932d455dac365f8ed9c5f699: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 18:33:30 dist-test-slave-cbn3 krb5kdc[21070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.1@KRBTEST.COM
I20260620 18:33:30.964943 21615 ts_tablet_manager.cc:1918] T 786611b0db5b4e1ea3b496a930b2a177 P 4052766d900c481887e1b157069b8fbc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:33:30.965209 21615 log.cc:1196] T 786611b0db5b4e1ea3b496a930b2a177 P 4052766d900c481887e1b157069b8fbc: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-2/wal/wals/786611b0db5b4e1ea3b496a930b2a177
I20260620 18:33:30.965742 21613 ts_tablet_manager.cc:1918] T 786611b0db5b4e1ea3b496a930b2a177 P f2e5540a932d455dac365f8ed9c5f699: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:33:30.965884 21613 log.cc:1196] T 786611b0db5b4e1ea3b496a930b2a177 P f2e5540a932d455dac365f8ed9c5f699: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-0/wal/wals/786611b0db5b4e1ea3b496a930b2a177
I20260620 18:33:30.966066 21615 ts_tablet_manager.cc:1939] T 786611b0db5b4e1ea3b496a930b2a177 P 4052766d900c481887e1b157069b8fbc: Deleting consensus metadata
I20260620 18:33:30.966336 21613 ts_tablet_manager.cc:1939] T 786611b0db5b4e1ea3b496a930b2a177 P f2e5540a932d455dac365f8ed9c5f699: Deleting consensus metadata
I20260620 18:33:30.968088 21100 catalog_manager.cc:4841] TS 4052766d900c481887e1b157069b8fbc (127.10.131.3:37711): tablet 786611b0db5b4e1ea3b496a930b2a177 (table test-table [id=58eeade460cc4a39b5ddd57daecae08c]) successfully deleted
I20260620 18:33:30.968432 21099 catalog_manager.cc:4841] TS f2e5540a932d455dac365f8ed9c5f699 (127.10.131.1:38675): tablet 786611b0db5b4e1ea3b496a930b2a177 (table test-table [id=58eeade460cc4a39b5ddd57daecae08c]) successfully deleted
I20260620 18:33:30.970136 21585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:30.945802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50908 (local address 127.10.131.1:38675)
0620 18:33:30.945984 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:30.945993 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:30.948804 (+ 2811us) server_negotiation.cc:407] Connection header received
0620 18:33:30.949109 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:30.949117 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:30.949241 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:30.949335 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:30.951610 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:30.953119 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:30.955055 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:30.955720 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:30.955847 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:30.961427 (+ 5580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:30.961442 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:30.961460 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:33:30.961506 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:30.965583 (+ 4077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:30.966857 (+ 1274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:30.966866 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:30.966869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:30.967042 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:30.967883 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:30.967888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:30.967891 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:30.968361 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:30.968638 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:30.969289 (+ 651us) server_negotiation.cc:299] Negotiation successful
0620 18:33:30.969491 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:33:30.971239 21299 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:33:30.981027 21614 tablet_replica.cc:330] T 786611b0db5b4e1ea3b496a930b2a177 P a75d97b23ea04dd485e5f597fc156bdd: stopping tablet replica
I20260620 18:33:30.981554 21614 raft_consensus.cc:2229] T 786611b0db5b4e1ea3b496a930b2a177 P a75d97b23ea04dd485e5f597fc156bdd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:33:30.982064 21614 raft_consensus.cc:2258] T 786611b0db5b4e1ea3b496a930b2a177 P a75d97b23ea04dd485e5f597fc156bdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:33:30.984390 21614 ts_tablet_manager.cc:1905] T 786611b0db5b4e1ea3b496a930b2a177 P a75d97b23ea04dd485e5f597fc156bdd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:33:30.991868 21614 ts_tablet_manager.cc:1918] T 786611b0db5b4e1ea3b496a930b2a177 P a75d97b23ea04dd485e5f597fc156bdd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:33:30.992053 21614 log.cc:1196] T 786611b0db5b4e1ea3b496a930b2a177 P a75d97b23ea04dd485e5f597fc156bdd: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781980314095189-10764-0/minicluster-data/ts-1/wal/wals/786611b0db5b4e1ea3b496a930b2a177
I20260620 18:33:30.992600 21614 ts_tablet_manager.cc:1939] T 786611b0db5b4e1ea3b496a930b2a177 P a75d97b23ea04dd485e5f597fc156bdd: Deleting consensus metadata
I20260620 18:33:30.994720 21101 catalog_manager.cc:4841] TS a75d97b23ea04dd485e5f597fc156bdd (127.10.131.2:39379): tablet 786611b0db5b4e1ea3b496a930b2a177 (table test-table [id=58eeade460cc4a39b5ddd57daecae08c]) successfully deleted
I20260620 18:33:31.076452 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 21176
I20260620 18:33:31.099439 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 21310
Jun 20 18:33:31 dist-test-slave-cbn3 krb5kdc[21070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.2@KRBTEST.COM
I20260620 18:33:31.118570 21587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:31.100508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53620 (local address 127.10.131.2:39379)
0620 18:33:31.100691 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:31.100697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:31.100717 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:33:31.100821 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:31.100826 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:31.100966 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:31.101076 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:31.102650 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:31.103809 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:31.105688 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:31.106354 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:31.106487 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:31.112035 (+ 5548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:31.112048 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:31.112067 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:33:31.112112 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:31.115864 (+ 3752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:31.116641 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:31.116647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:31.116650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:31.116760 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:31.117194 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:31.117198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:31.117200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:31.117552 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:31.117757 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:31.118107 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 18:33:31.118281 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:33:31.119320 21433 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:33:31.207792 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 21310
I20260620 18:33:31.225265 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 21444
Jun 20 18:33:31 dist-test-slave-cbn3 krb5kdc[21070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.3@KRBTEST.COM
I20260620 18:33:31.243019 21589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:33:31.226114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56208 (local address 127.10.131.3:37711)
0620 18:33:31.226320 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 18:33:31.226328 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:33:31.226349 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:33:31.226570 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:33:31.226576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:33:31.226658 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:33:31.226788 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:33:31.228056 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:31.229269 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:31.230482 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:33:31.231273 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:33:31.231458 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:33:31.235904 (+ 4446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:33:31.235921 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:33:31.235943 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:33:31.235996 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:33:31.240003 (+ 4007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:31.240765 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:31.240778 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:31.240782 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:31.240985 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:33:31.241334 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:33:31.241339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:33:31.241343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:33:31.241812 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:33:31.242161 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0620 18:33:31.242320 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 18:33:31.242546 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:33:31.243947 21567 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:33:31.381736 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 21444
I20260620 18:33:31.400992 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 21083
I20260620 18:33:31.402102 21145 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:33:31.509238 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 21083
2026-06-20T18:33:31Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4344 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:33:31 dist-test-slave-cbn3 krb5kdc[21623](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:33:31 dist-test-slave-cbn3 krb5kdc[21623](info): set up 2 sockets
Jun 20 18:33:31 dist-test-slave-cbn3 krb5kdc[21623](info): commencing operation
krb5kdc: starting...
W20260620 18:33:32.605784 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:33:32 dist-test-slave-cbn3 krb5kdc[21623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:33:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:33:32Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.10.131.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.10.131.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.10.131.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/rangeradmin_127.10.131.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.10.131.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/rangeradmin_127.10.131.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.10.131.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.10.131.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.10.131.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/rangerlookup_127.10.131.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.10.131.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/rangerlookup_127.10.131.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.131.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.131.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.131.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/HTTP_127.10.131.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.131.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/HTTP_127.10.131.20@KRBTEST.COM.keytab.
I20260620 18:33:32.916038 10764 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task0dhafo/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/postgres -l logfile start
2026-06-20 18:33:35.232 UTC [21653] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 18:33:35.232 UTC [21653] LOG: listening on IPv4 address "127.10.131.20", port 45787
2026-06-20 18:33:35.235 UTC [21653] LOG: listening on Unix socket "/tmp/.s.PGSQL.45787"
2026-06-20 18:33:35.259 UTC [21656] LOG: database system was shut down at 2026-06-20 18:33:33 UTC
2026-06-20 18:33:35.263 UTC [21653] LOG: database system is ready to accept connections
I20260620 18:33:36.229456 10764 mini_postgres.cc:96] Postgres bound to 45787
2026-06-20 18:33:36.248 UTC [21664] FATAL: database "slave" does not exist
127.10.131.20:45787 - accepting connections
I20260620 18:33:36.248960 10764 mini_ranger.cc:161] Starting Ranger...
I20260620 18:33:36.277029 10764 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 18:33:36.700675 10764 mini_ranger.cc:87] Created ranger Postgres database
I20260620 18:33:36.700806 10764 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ranger-admin
2026-06-20 18:33:36,986 [I] DB FLAVOR :POSTGRES
2026-06-20 18:33:36,986 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 18:33:36,986 [I] Checking connection..
2026-06-20 18:33:36,986 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:45787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 18:33:37,238 [I] Checking connection passed.
2026-06-20 18:33:37,238 [I] --------- Verifying version history table ---------
2026-06-20 18:33:37,239 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:45787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:33:37,475 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 18:33:37,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:45787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:33:37,734 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 18:33:37,734 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 18:33:37,734 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:45787/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 18:33:37.980 UTC [21728] WARNING: there is no transaction in progress
2026-06-20 18:33:37,986 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 18:33:37,986 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:45787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:33:38,221 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 18:33:38,221 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 18:33:38,222 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:45787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 18:33:38,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:45787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 18:33:38,671 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:45787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-cbn3.c.gcp-upstream.internal','N') ;"
2026-06-20 18:33:38,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:45787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 18:33:39,154 [I] Table x_portal_user does not exist in database ranger
2026-06-20 18:33:39,155 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:45787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 18:33:39,384 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 18:33:39,385 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:45787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 18:33:39,641 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:45787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 18:33:39,900 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 18:33:39,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:45787/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 18:33:40.817 UTC [21836] WARNING: there is no transaction in progress
2026-06-20 18:33:40.832 UTC [21836] WARNING: there is no transaction in progress
2026-06-20 18:33:40.848 UTC [21836] WARNING: there is no transaction in progress
2026-06-20 18:33:40.861 UTC [21836] WARNING: there is no transaction in progress
2026-06-20 18:33:40.875 UTC [21836] WARNING: there is no transaction in progress
2026-06-20 18:33:40.892 UTC [21836] WARNING: there is no transaction in progress
2026-06-20 18:33:40.953 UTC [21836] WARNING: there is no transaction in progress
2026-06-20 18:33:40.959 UTC [21836] WARNING: there is no transaction in progress
2026-06-20 18:33:40.972 UTC [21836] WARNING: there is no transaction in progress
2026-06-20 18:33:40.981 UTC [21836] WARNING: there is no transaction in progress
2026-06-20 18:33:40.991 UTC [21836] WARNING: there is no transaction in progress
2026-06-20 18:33:41.003 UTC [21836] WARNING: there is no transaction in progress
2026-06-20 18:33:41.012 UTC [21836] WARNING: there is no transaction in progress
2026-06-20 18:33:41.018 UTC [21836] WARNING: there is no transaction in progress
2026-06-20 18:33:41.024 UTC [21836] WARNING: there is no transaction in progress
2026-06-20 18:33:41.030 UTC [21836] WARNING: there is no transaction in progress
2026-06-20 18:33:41,588 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 18:33:41,588 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:45787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 18:33:41,810 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 18:33:41,811 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:45787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 18:33:42,046 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 18:33:42,046 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:45787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 18:33:42,276 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 18:33:42,277 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:45787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 18:33:42,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:45787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 18:33:42,777 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:45787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-cbn3.c.gcp-upstream.internal';"
2026-06-20 18:33:43,027 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 18:33:43,027 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:45787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 18:33:43,284 [I] DB_PATCHES have already been applied
I20260620 18:33:43.290424 10764 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ranger-admin:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task0dhafo/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 18:33:43.290514 10764 mini_ranger.cc:193] Using host: 127.10.131.20
I20260620 18:33:43.305339 10764 mini_ranger.cc:227] Ranger admin URL: http://127.10.131.20:59249
Jun 20, 2026 6:33:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:33:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:33:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 6:33:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 6:33:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 6:33:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 6:33:44 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 18:33:44.427577 10764 mini_ranger.cc:160] Time spent starting Ranger: real 8.179s user 0.004s sys 0.036s
Jun 20, 2026 6:34:04 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 6:34:04 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 6:34:04 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 6:34:04 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 6:34:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 6:34:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 6:34:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 6:34:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 6:34:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 6:34:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 6:34:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 6:34:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 6:34:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 6:34:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 6:34:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 6:34:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 6:34:04 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 6:34:05 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 18:34:06.469302 10764 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.10.131.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.10.131.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.10.131.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/rangerkms_127.10.131.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.10.131.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/rangerkms_127.10.131.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.131.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.10.131.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.131.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/HTTP_127.10.131.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.131.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/HTTP_127.10.131.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 18:34:06 dist-test-slave-cbn3 krb5kdc[21623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980446, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 18:34:06.644615 10764 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 18:34:06.667907 10764 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 18:34:07.387341 10764 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 18:34:07.387506 10764 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ranger-kms
I20260620 18:34:07.387658 10764 mini_ranger_kms.cc:227] Using postgres at 127.10.131.20:45787
2026-06-20 18:34:07,826 [I] DB FLAVOR :POSTGRES
2026-06-20 18:34:07,826 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 18:34:07,826 [I] Checking connection
2026-06-20 18:34:07,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:45787/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 18:34:08,065 [I] connection success
2026-06-20 18:34:08,065 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 18:34:08,065 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 18:34:08,065 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:45787/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 18:34:08,293 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 18:34:08,293 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 18:34:08,293 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 18:34:08,293 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:45787/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 18:34:08,548 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 18:34:09.006630 10764 mini_ranger_kms.cc:310] Created kms service
I20260620 18:34:09.176532 10764 mini_ranger_kms.cc:326] Created kudu user
I20260620 18:34:09.219349 10764 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 18:34:09 dist-test-slave-cbn3 krb5kdc[21623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980446, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.10.131.20@KRBTEST.COM
I20260620 18:34:09.617105 10764 mini_ranger_kms.cc:384] Added ranger policy
I20260620 18:34:09.617444 10764 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ranger-kms:/tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 18:34:09.617511 10764 mini_ranger_kms.cc:240] Using host: 127.10.131.20
I20260620 18:34:09.637049 10764 mini_ranger_kms.cc:276] Ranger KMS PID: 22016
I20260620 18:34:09.637250 10764 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.10.131.20:34925
Jun 20, 2026 6:34:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:34:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:34:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 6:34:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 6:34:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 6:34:11 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 18:34:11.937045 10764 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.292s user 0.138s sys 0.325s
I20260620 18:34:11.937319 10764 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 18:34:11.937549 10764 mini_ranger_kms.cc:401] 127.10.131.20:34925/kms/v1/keys
Jun 20 18:34:12 dist-test-slave-cbn3 krb5kdc[21623](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781980452, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.131.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 18:34:12 dist-test-slave-cbn3 krb5kdc[21623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980452, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.131.20@KRBTEST.COM for HTTP/127.10.131.20@KRBTEST.COM
Jun 20 18:34:12 dist-test-slave-cbn3 krb5kdc[21623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980452, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.131.20@KRBTEST.COM for HTTP/127.10.131.20@KRBTEST.COM
Jun 20 18:34:14 dist-test-slave-cbn3 krb5kdc[21623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.10.131.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
I20260620 18:34:15.054529 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:43609
--webserver_interface=127.10.131.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:43555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:43609
--ranger_config_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.131.20:34925/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:34:15.391292 22067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:15.391626 22067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:15.391767 22067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:15.399089 22067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:34:15.399233 22067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:15.399324 22067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:15.399421 22067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:34:15.399495 22067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:34:15.409685 22067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:43555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/master-0/wal
--ranger_kms_url=127.10.131.20:34925/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:43609
--ranger_config_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:43609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:34:15.410985 22067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:15.412252 22067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:15.423960 22075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:15.423941 22073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:15.424255 22072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:15.424688 22067 server_base.cc:1034] running on GCE node
I20260620 18:34:15.425611 22067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:15.427047 22067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:15.428297 22067 hybrid_clock.cc:648] HybridClock initialized: now 1781980455428255 us; error 37 us; skew 500 ppm
I20260620 18:34:15.428625 22067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:34:15 dist-test-slave-cbn3 krb5kdc[21623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980455, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:15.431838 22067 init.cc:377] Logged in from keytab as kudu/127.10.131.62@KRBTEST.COM (short username kudu)
I20260620 18:34:15.433491 22067 webserver.cc:460] Webserver started at http://127.10.131.62:33723/ using document root <none> and password file <none>
I20260620 18:34:15.434094 22067 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:15.434255 22067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:15.434537 22067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:34:15 dist-test-slave-cbn3 krb5kdc[21623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980455, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for HTTP/127.10.131.20@KRBTEST.COM
I20260620 18:34:15.488953 22067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "9bd1dafd754c42689d70bbb7227f700a"
format_stamp: "Formatted at 2026-06-20 18:34:15 on dist-test-slave-cbn3"
server_key: "39ccefebeb6d995377eb5424472f113a"
server_key_iv: "8a55322a1de32fbf30cf51b96c1168e9"
server_key_version: "kuduclusterkey@0"
I20260620 18:34:15.490027 22067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "9bd1dafd754c42689d70bbb7227f700a"
format_stamp: "Formatted at 2026-06-20 18:34:15 on dist-test-slave-cbn3"
server_key: "39ccefebeb6d995377eb5424472f113a"
server_key_iv: "8a55322a1de32fbf30cf51b96c1168e9"
server_key_version: "kuduclusterkey@0"
I20260620 18:34:15.496646 22067 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 18:34:15.513958 22082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:15.515458 22067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:34:15.515635 22067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "9bd1dafd754c42689d70bbb7227f700a"
format_stamp: "Formatted at 2026-06-20 18:34:15 on dist-test-slave-cbn3"
server_key: "39ccefebeb6d995377eb5424472f113a"
server_key_iv: "8a55322a1de32fbf30cf51b96c1168e9"
server_key_version: "kuduclusterkey@0"
I20260620 18:34:15.515863 22067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:15.537904 22067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:15.541874 22067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:15.542176 22067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:15.561295 22067 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:43609
I20260620 18:34:15.561318 22133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:43609 every 8 connection(s)
I20260620 18:34:15.562604 22067 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:34:15.563076 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 22067
I20260620 18:34:15.563270 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:34:15.564035 10764 external_mini_cluster.cc:1428] Setting key 13e6c5c1c147b3795dc17e0e6d053b10
I20260620 18:34:15.567941 22134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:34:15 dist-test-slave-cbn3 krb5kdc[21623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:34:15.585304 22134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bd1dafd754c42689d70bbb7227f700a: Bootstrap starting.
I20260620 18:34:15.588819 22137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:15.566524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48470 (local address 127.10.131.62:43609)
0620 18:34:15.569448 (+ 2924us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:15.569461 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:15.569507 (+ 46us) server_negotiation.cc:407] Connection header received
0620 18:34:15.572346 (+ 2839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:15.572384 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:15.573048 (+ 664us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:15.573463 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:15.574801 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:15.576120 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:15.577404 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:15.578055 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:15.578252 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:15.582021 (+ 3769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:15.582029 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:15.582043 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:15.582076 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:15.584965 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:15.585698 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:15.585706 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:15.585711 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:15.585863 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:15.586222 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:15.586228 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:15.586231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:15.586707 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:15.587004 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:15.587474 (+ 470us) server_negotiation.cc:299] Negotiation successful
0620 18:34:15.587893 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2703,"thread_start_us":2481,"threads_started":1}
I20260620 18:34:15.590500 22134 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bd1dafd754c42689d70bbb7227f700a: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:15.591643 22134 log.cc:824] T 00000000000000000000000000000000 P 9bd1dafd754c42689d70bbb7227f700a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:15.594322 22134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bd1dafd754c42689d70bbb7227f700a: No bootstrap required, opened a new log
I20260620 18:34:15.601867 22134 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bd1dafd754c42689d70bbb7227f700a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd1dafd754c42689d70bbb7227f700a" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 43609 } }
I20260620 18:34:15.602283 22134 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bd1dafd754c42689d70bbb7227f700a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:15.602394 22134 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bd1dafd754c42689d70bbb7227f700a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bd1dafd754c42689d70bbb7227f700a, State: Initialized, Role: FOLLOWER
I20260620 18:34:15.602959 22134 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bd1dafd754c42689d70bbb7227f700a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd1dafd754c42689d70bbb7227f700a" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 43609 } }
I20260620 18:34:15.603224 22134 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bd1dafd754c42689d70bbb7227f700a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:34:15.603451 22134 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bd1dafd754c42689d70bbb7227f700a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:34:15.603679 22134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bd1dafd754c42689d70bbb7227f700a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:15.607499 22134 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bd1dafd754c42689d70bbb7227f700a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd1dafd754c42689d70bbb7227f700a" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 43609 } }
I20260620 18:34:15.608269 22134 leader_election.cc:302] T 00000000000000000000000000000000 P 9bd1dafd754c42689d70bbb7227f700a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bd1dafd754c42689d70bbb7227f700a; no voters:
I20260620 18:34:15.609086 22134 leader_election.cc:288] T 00000000000000000000000000000000 P 9bd1dafd754c42689d70bbb7227f700a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:34:15.609191 22139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bd1dafd754c42689d70bbb7227f700a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:15.612082 22134 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bd1dafd754c42689d70bbb7227f700a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:34:15.609944 22139 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bd1dafd754c42689d70bbb7227f700a [term 1 LEADER]: Becoming Leader. State: Replica: 9bd1dafd754c42689d70bbb7227f700a, State: Running, Role: LEADER
I20260620 18:34:15.613487 22139 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bd1dafd754c42689d70bbb7227f700a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd1dafd754c42689d70bbb7227f700a" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 43609 } }
I20260620 18:34:15.616871 22140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bd1dafd754c42689d70bbb7227f700a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bd1dafd754c42689d70bbb7227f700a. Latest consensus state: current_term: 1 leader_uuid: "9bd1dafd754c42689d70bbb7227f700a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd1dafd754c42689d70bbb7227f700a" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 43609 } } }
I20260620 18:34:15.616780 22141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bd1dafd754c42689d70bbb7227f700a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bd1dafd754c42689d70bbb7227f700a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd1dafd754c42689d70bbb7227f700a" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 43609 } } }
I20260620 18:34:15.617401 22140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bd1dafd754c42689d70bbb7227f700a [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:15.617416 22141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bd1dafd754c42689d70bbb7227f700a [sys.catalog]: This master's current role is: LEADER
I20260620 18:34:15.626130 22147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:34:15.630252 22147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:34:15.630959 22134 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 18:34:15.646704 22147 catalog_manager.cc:1347] Generated new cluster ID: 65d20198e9d44510981273acb22038e9
I20260620 18:34:15.646858 22147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:34:15.660036 22147 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:34:15.662007 22147 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:34:15.672968 22147 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bd1dafd754c42689d70bbb7227f700a: Generated new TSK 0
I20260620 18:34:15.674144 22147 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.131.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab.
Entry for principal kudu/127.10.131.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab.
I20260620 18:34:16.442894 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.131.20:34925/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.1:0
--local_ip_for_outbound_sockets=127.10.131.1
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:43609
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:43555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:34:16 dist-test-slave-cbn3 krb5kdc[21623](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781980456, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 18:34:16.887575 22179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:16.887976 22179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:16.888144 22179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:16.900295 22179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:16.900470 22179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:16.900676 22179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.1
I20260620 18:34:16.916889 22179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:43555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.10.131.20:34925/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:43609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.1
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:34:16.918644 22179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:16.920459 22179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:16.937664 22189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:16.941601 22190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:16.950434 22179 server_base.cc:1034] running on GCE node
W20260620 18:34:16.951274 22193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:16.952203 22179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:16.953377 22179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:16.954603 22179 hybrid_clock.cc:648] HybridClock initialized: now 1781980456954540 us; error 57 us; skew 500 ppm
I20260620 18:34:16.954944 22179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:34:16 dist-test-slave-cbn3 krb5kdc[21623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980456, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:16.962426 22179 init.cc:377] Logged in from keytab as kudu/127.10.131.1@KRBTEST.COM (short username kudu)
I20260620 18:34:16.964742 22179 webserver.cc:460] Webserver started at http://127.10.131.1:42449/ using document root <none> and password file <none>
I20260620 18:34:16.965490 22179 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:16.965627 22179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:16.965955 22179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:34:16 dist-test-slave-cbn3 krb5kdc[21623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980456, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for HTTP/127.10.131.20@KRBTEST.COM
I20260620 18:34:16.989535 22179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-0/data/instance:
uuid: "01c58f5ea6e94355ab61a3b194f01307"
format_stamp: "Formatted at 2026-06-20 18:34:16 on dist-test-slave-cbn3"
server_key: "8e4ec08525f1b8158c273fd2c8439fc7"
server_key_iv: "f44df4867210ed9523e07a14ac2229e8"
server_key_version: "kuduclusterkey@0"
I20260620 18:34:16.990746 22179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance:
uuid: "01c58f5ea6e94355ab61a3b194f01307"
format_stamp: "Formatted at 2026-06-20 18:34:16 on dist-test-slave-cbn3"
server_key: "8e4ec08525f1b8158c273fd2c8439fc7"
server_key_iv: "f44df4867210ed9523e07a14ac2229e8"
server_key_version: "kuduclusterkey@0"
I20260620 18:34:17.002840 22179 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.007s sys 0.004s
I20260620 18:34:17.045526 22199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:17.047578 22179 fs_manager.cc:725] Time spent opening block manager: real 0.023s user 0.007s sys 0.015s
I20260620 18:34:17.047763 22179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-0/wal
uuid: "01c58f5ea6e94355ab61a3b194f01307"
format_stamp: "Formatted at 2026-06-20 18:34:16 on dist-test-slave-cbn3"
server_key: "8e4ec08525f1b8158c273fd2c8439fc7"
server_key_iv: "f44df4867210ed9523e07a14ac2229e8"
server_key_version: "kuduclusterkey@0"
I20260620 18:34:17.048046 22179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:17.072712 22179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:17.078017 22179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:17.078344 22179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:17.085175 22179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:17.087110 22179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:17.087210 22179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:17.087342 22179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:17.087405 22179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:17.132259 22179 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.1:36351
I20260620 18:34:17.132287 22311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.1:36351 every 8 connection(s)
I20260620 18:34:17.134094 22179 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
I20260620 18:34:17.141943 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 22179
I20260620 18:34:17.142212 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance
I20260620 18:34:17.143179 10764 external_mini_cluster.cc:1428] Setting key a464eaaf0fdb923fa60d15f8e269b5ed
Jun 20 18:34:17 dist-test-slave-cbn3 krb5kdc[21623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980456, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:34:17.167339 22314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:17.137540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:58691 (local address 127.10.131.62:43609)
0620 18:34:17.138021 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:17.138027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:17.145433 (+ 7406us) server_negotiation.cc:407] Connection header received
0620 18:34:17.148068 (+ 2635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:17.148074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:17.148171 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:17.148261 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:17.150559 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:17.151697 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:17.154225 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:17.154866 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:17.154969 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:17.159943 (+ 4974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:17.159949 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:17.159954 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:17.160001 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:17.163571 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:17.164483 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:17.164489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:17.164492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:17.164643 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:17.165258 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:17.165263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:17.165266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:17.165600 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:17.165727 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:17.166727 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0620 18:34:17.166963 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":243,"threads_started":1}
I20260620 18:34:17.169718 22315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:17.137802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:43609 (local address 127.10.131.1:58691)
0620 18:34:17.145248 (+ 7446us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:17.145305 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:17.147727 (+ 2422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:17.148859 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:17.148875 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:17.149514 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:17.150344 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:17.150370 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:17.151913 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:17.151918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:17.154053 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:17.154070 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:17.155294 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:17.155301 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:17.155596 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:17.156410 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:17.156440 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:17.159701 (+ 3261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:17.163748 (+ 4047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:17.163756 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:17.163774 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:17.164332 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:17.164788 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:17.164792 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:17.164796 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:17.165102 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:17.165823 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:17.165830 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:17.166282 (+ 452us) client_negotiation.cc:769] Sending connection context
0620 18:34:17.167823 (+ 1541us) client_negotiation.cc:240] Negotiation successful
0620 18:34:17.168352 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7153,"thread_start_us":6947,"threads_started":1}
WARNING: no policy specified for kudu/127.10.131.2@KRBTEST.COM; defaulting to no policy
I20260620 18:34:17.172071 22312 heartbeater.cc:347] Connected to a master server at 127.10.131.62:43609
I20260620 18:34:17.172402 22312 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:17.173216 22312 heartbeater.cc:511] Master 127.10.131.62:43609 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.2@KRBTEST.COM" created.
I20260620 18:34:17.175594 22099 ts_manager.cc:194] Registered new tserver with Master: 01c58f5ea6e94355ab61a3b194f01307 (127.10.131.1:36351)
I20260620 18:34:17.177438 22099 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.1@KRBTEST.COM'} at 127.10.131.1:58691
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab.
Entry for principal kudu/127.10.131.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab.
I20260620 18:34:17.209436 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.131.20:34925/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.2:0
--local_ip_for_outbound_sockets=127.10.131.2
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:43609
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:43555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:34:17 dist-test-slave-cbn3 krb5kdc[21623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980456, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for HTTP/127.10.131.20@KRBTEST.COM
Jun 20 18:34:17 dist-test-slave-cbn3 krb5kdc[21623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980456, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for HTTP/127.10.131.20@KRBTEST.COM
W20260620 18:34:17.726537 22318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:17.727082 22318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:17.727316 22318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:17.741110 22318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:17.741330 22318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:17.741611 22318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.2
I20260620 18:34:17.758908 22318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:43555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.10.131.20:34925/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:43609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.2
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:34:17.760800 22318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:17.762606 22318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:17.780392 22325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:17.791520 22326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:17.793215 22328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:17.793880 22318 server_base.cc:1034] running on GCE node
I20260620 18:34:17.794653 22318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:17.796176 22318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:17.797490 22318 hybrid_clock.cc:648] HybridClock initialized: now 1781980457797379 us; error 87 us; skew 500 ppm
I20260620 18:34:17.797938 22318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:34:17 dist-test-slave-cbn3 krb5kdc[21623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980457, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:17.802870 22318 init.cc:377] Logged in from keytab as kudu/127.10.131.2@KRBTEST.COM (short username kudu)
I20260620 18:34:17.818923 22318 webserver.cc:460] Webserver started at http://127.10.131.2:35285/ using document root <none> and password file <none>
I20260620 18:34:17.820104 22318 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:17.820266 22318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:17.820595 22318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:34:17 dist-test-slave-cbn3 krb5kdc[21623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980457, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for HTTP/127.10.131.20@KRBTEST.COM
I20260620 18:34:17.844466 22318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-1/data/instance:
uuid: "9a720b4005b041f4803fe0fefff4650b"
format_stamp: "Formatted at 2026-06-20 18:34:17 on dist-test-slave-cbn3"
server_key: "b85fcb3fe0a0e6d42d8c86b9ab895836"
server_key_iv: "095fde98960bceeadc4b58dec8e2403e"
server_key_version: "kuduclusterkey@0"
I20260620 18:34:17.845634 22318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance:
uuid: "9a720b4005b041f4803fe0fefff4650b"
format_stamp: "Formatted at 2026-06-20 18:34:17 on dist-test-slave-cbn3"
server_key: "b85fcb3fe0a0e6d42d8c86b9ab895836"
server_key_iv: "095fde98960bceeadc4b58dec8e2403e"
server_key_version: "kuduclusterkey@0"
I20260620 18:34:17.853032 22318 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260620 18:34:17.872386 22342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:17.874398 22318 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260620 18:34:17.874581 22318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-1/wal
uuid: "9a720b4005b041f4803fe0fefff4650b"
format_stamp: "Formatted at 2026-06-20 18:34:17 on dist-test-slave-cbn3"
server_key: "b85fcb3fe0a0e6d42d8c86b9ab895836"
server_key_iv: "095fde98960bceeadc4b58dec8e2403e"
server_key_version: "kuduclusterkey@0"
I20260620 18:34:17.874851 22318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:17.894805 22318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:17.900101 22318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:17.900460 22318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:17.901693 22318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:17.903528 22318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:17.903635 22318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:17.903811 22318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:17.903885 22318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:17.946326 22318 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.2:34939
I20260620 18:34:17.946326 22454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.2:34939 every 8 connection(s)
I20260620 18:34:17.948276 22318 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
I20260620 18:34:17.952527 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 22318
I20260620 18:34:17.952692 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance
I20260620 18:34:17.953336 10764 external_mini_cluster.cc:1428] Setting key 9275e115ca8accfe07a6ac9381a3721c
WARNING: no policy specified for kudu/127.10.131.3@KRBTEST.COM; defaulting to no policy
Jun 20 18:34:17 dist-test-slave-cbn3 krb5kdc[21623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980457, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.3@KRBTEST.COM" created.
I20260620 18:34:18.003417 22457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:17.951282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:38591 (local address 127.10.131.62:43609)
0620 18:34:17.970240 (+ 18958us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:17.970252 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:17.971541 (+ 1289us) server_negotiation.cc:407] Connection header received
0620 18:34:17.975583 (+ 4042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:17.975590 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:17.975693 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:17.975808 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:17.981406 (+ 5598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:17.982704 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:17.985725 (+ 3021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:17.986520 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:17.986656 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:17.991918 (+ 5262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:17.991923 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:17.991927 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:17.991971 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:17.998723 (+ 6752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:17.999875 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:17.999884 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:17.999891 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:18.000054 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:18.000789 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:18.000797 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:18.000803 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:18.001260 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:18.001417 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:18.002585 (+ 1168us) server_negotiation.cc:299] Negotiation successful
0620 18:34:18.002867 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18737,"thread_start_us":20236,"threads_started":1}
I20260620 18:34:18.006228 22458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:17.951802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:43609 (local address 127.10.131.2:38591)
0620 18:34:17.971114 (+ 19312us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:17.971196 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:17.975262 (+ 4066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:17.979352 (+ 4090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:17.979367 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:17.980197 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:17.981167 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:17.981200 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:17.982952 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:17.982958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:17.985519 (+ 2561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:17.985539 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:17.987074 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:17.987085 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:17.987444 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:17.988312 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:17.988356 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:17.991522 (+ 3166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:17.998955 (+ 7433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:17.998969 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:17.998992 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:17.999712 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:18.000224 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:18.000231 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:18.000238 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:18.000597 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:18.001542 (+ 945us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:18.001553 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:18.002080 (+ 527us) client_negotiation.cc:769] Sending connection context
0620 18:34:18.004035 (+ 1955us) client_negotiation.cc:240] Negotiation successful
0620 18:34:18.004687 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18855,"thread_start_us":18585,"threads_started":1}
I20260620 18:34:18.008299 22455 heartbeater.cc:347] Connected to a master server at 127.10.131.62:43609
I20260620 18:34:18.008656 22455 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:18.009601 22455 heartbeater.cc:511] Master 127.10.131.62:43609 requested a full tablet report, sending...
I20260620 18:34:18.011555 22099 ts_manager.cc:194] Registered new tserver with Master: 9a720b4005b041f4803fe0fefff4650b (127.10.131.2:34939)
I20260620 18:34:18.012938 22099 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.2@KRBTEST.COM'} at 127.10.131.2:38591
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab.
Entry for principal kudu/127.10.131.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab.
I20260620 18:34:18.025354 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.131.20:34925/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.3:0
--local_ip_for_outbound_sockets=127.10.131.3
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:43609
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:43555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:34:18.180791 22312 heartbeater.cc:503] Master 127.10.131.62:43609 was elected leader, sending a full tablet report...
W20260620 18:34:18.358271 22461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:34:18.358518 22461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:34:18.358614 22461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:34:18.366009 22461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:34:18.366106 22461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:34:18.366210 22461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.3
I20260620 18:34:18.375644 22461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:43555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.10.131.20:34925/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:43609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.3
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:34:18.376948 22461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:34:18.378089 22461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:34:18.391402 22466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:18.391347 22469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:34:18.391382 22467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:34:18.391940 22461 server_base.cc:1034] running on GCE node
I20260620 18:34:18.392673 22461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:34:18.393611 22461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:34:18.394836 22461 hybrid_clock.cc:648] HybridClock initialized: now 1781980458394781 us; error 51 us; skew 500 ppm
I20260620 18:34:18.395135 22461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:34:18 dist-test-slave-cbn3 krb5kdc[21623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980458, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:34:18.398231 22461 init.cc:377] Logged in from keytab as kudu/127.10.131.3@KRBTEST.COM (short username kudu)
I20260620 18:34:18.399515 22461 webserver.cc:460] Webserver started at http://127.10.131.3:37951/ using document root <none> and password file <none>
I20260620 18:34:18.400125 22461 fs_manager.cc:359] Metadata directory not provided
I20260620 18:34:18.400220 22461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:34:18.400461 22461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:34:18 dist-test-slave-cbn3 krb5kdc[21623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980458, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for HTTP/127.10.131.20@KRBTEST.COM
I20260620 18:34:18.414355 22461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-2/data/instance:
uuid: "589b9774bfdd47bf850f2da794ce8296"
format_stamp: "Formatted at 2026-06-20 18:34:18 on dist-test-slave-cbn3"
server_key: "a5265fdae56b2197488deefd37b263b8"
server_key_iv: "95924b86efa09468daa9196d25e20920"
server_key_version: "kuduclusterkey@0"
I20260620 18:34:18.415138 22461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance:
uuid: "589b9774bfdd47bf850f2da794ce8296"
format_stamp: "Formatted at 2026-06-20 18:34:18 on dist-test-slave-cbn3"
server_key: "a5265fdae56b2197488deefd37b263b8"
server_key_iv: "95924b86efa09468daa9196d25e20920"
server_key_version: "kuduclusterkey@0"
I20260620 18:34:18.420480 22461 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:34:18.436532 22476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:18.438084 22461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:34:18.438261 22461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-2/wal
uuid: "589b9774bfdd47bf850f2da794ce8296"
format_stamp: "Formatted at 2026-06-20 18:34:18 on dist-test-slave-cbn3"
server_key: "a5265fdae56b2197488deefd37b263b8"
server_key_iv: "95924b86efa09468daa9196d25e20920"
server_key_version: "kuduclusterkey@0"
I20260620 18:34:18.438455 22461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:34:18.455403 22461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:34:18.459309 22461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:34:18.459585 22461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:34:18.460453 22461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:34:18.462144 22461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:34:18.462215 22461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:18.462391 22461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:34:18.462442 22461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:34:18.496237 22461 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.3:37057
I20260620 18:34:18.496279 22588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.3:37057 every 8 connection(s)
I20260620 18:34:18.497498 22461 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
I20260620 18:34:18.497874 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 22461
I20260620 18:34:18.498112 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance
I20260620 18:34:18.498731 10764 external_mini_cluster.cc:1428] Setting key 8f0c75f0cf410bbd62a7c4d71d984992
Jun 20 18:34:18 dist-test-slave-cbn3 krb5kdc[21623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980458, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:34:18.517571 22457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.500197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:40017 (local address 127.10.131.62:43609)
0620 18:34:18.500385 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:18.500390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:18.501948 (+ 1558us) server_negotiation.cc:407] Connection header received
0620 18:34:18.503673 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:18.503677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:18.503751 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:18.503832 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:18.505218 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.505907 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.507780 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.508214 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.508311 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.511454 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:18.511459 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:18.511462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:18.511492 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:18.514697 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:18.515376 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:18.515382 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:18.515385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:18.515504 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:18.515910 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:18.515915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:18.515918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:18.516177 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:18.516356 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:18.517004 (+ 648us) server_negotiation.cc:299] Negotiation successful
0620 18:34:18.517240 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:34:18.518391 22591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.500787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:43609 (local address 127.10.131.3:40017)
0620 18:34:18.501803 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:18.501846 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:18.503441 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:18.504031 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:18.504039 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:18.504449 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:34:18.505070 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.505089 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.506028 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.506032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.507586 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.507597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.508421 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.508438 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.508646 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.509323 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:34:18.509344 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:34:18.511254 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:34:18.514827 (+ 3573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:18.514836 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:18.514853 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:18.515258 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:18.515607 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:34:18.515613 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:34:18.515617 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:34:18.515796 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:34:18.516314 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:34:18.516321 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:34:18.516624 (+ 303us) client_negotiation.cc:769] Sending connection context
0620 18:34:18.516895 (+ 271us) client_negotiation.cc:240] Negotiation successful
0620 18:34:18.517351 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"thread_start_us":473,"threads_started":1}
I20260620 18:34:18.519608 22589 heartbeater.cc:347] Connected to a master server at 127.10.131.62:43609
I20260620 18:34:18.519902 22589 heartbeater.cc:464] Registering TS with master...
I20260620 18:34:18.520613 22589 heartbeater.cc:511] Master 127.10.131.62:43609 requested a full tablet report, sending...
I20260620 18:34:18.521916 22099 ts_manager.cc:194] Registered new tserver with Master: 589b9774bfdd47bf850f2da794ce8296 (127.10.131.3:37057)
I20260620 18:34:18.522683 22099 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.3@KRBTEST.COM'} at 127.10.131.3:40017
I20260620 18:34:18.526341 10764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:34:18.542578 22457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.530680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53950 (local address 127.10.131.62:43609)
0620 18:34:18.530870 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:18.530879 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:18.531257 (+ 378us) server_negotiation.cc:407] Connection header received
0620 18:34:18.531375 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:18.531378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:18.531439 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:18.531514 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:18.532768 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.533509 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.535224 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.535626 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.535738 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.537892 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:18.537896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:18.537899 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:18.537926 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:18.540295 (+ 2369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:18.540929 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:18.540933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:18.540935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:18.541050 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:18.541375 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:18.541378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:18.541380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:18.541677 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:18.541890 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:18.542130 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 18:34:18.542281 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:34:18.546281 22099 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:34:18.553409 22099 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:34:18.577214 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.565727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:37057 (local address 127.0.0.1:34640)
0620 18:34:18.567343 (+ 1616us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:18.567378 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:18.567596 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:18.568446 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:18.568456 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:18.568486 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:18.569413 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.569426 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.571634 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.571641 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.574002 (+ 2361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.574388 (+ 386us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.576259 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.576264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.576479 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.576539 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 18:34:18.576644 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 18:34:18.576821 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1367,"thread_start_us":1184,"threads_started":1}
I20260620 18:34:18.578552 22604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.566556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34640 (local address 127.10.131.3:37057)
0620 18:34:18.567313 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:18.567321 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:18.567494 (+ 173us) server_negotiation.cc:407] Connection header received
0620 18:34:18.567790 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:18.567800 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:18.568060 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:18.569602 (+ 1542us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:18.569876 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.571482 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.574585 (+ 3103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.576016 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.577745 (+ 1729us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.577970 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:18.578037 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 18:34:18.578192 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":445,"threads_started":1}
I20260620 18:34:18.580075 22603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.566051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:36351 (local address 127.0.0.1:39250)
0620 18:34:18.568144 (+ 2093us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:18.568165 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:18.568375 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:18.570735 (+ 2360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:18.570740 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:18.570763 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:18.571121 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.571131 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.574113 (+ 2982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.574118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.576252 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.576263 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.579301 (+ 3038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.579305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.579421 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.579441 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 18:34:18.579544 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 18:34:18.579734 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1971,"thread_start_us":1835,"threads_started":1}
I20260620 18:34:18.580138 22602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.566124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39250 (local address 127.10.131.1:36351)
0620 18:34:18.566753 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:18.566762 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:18.568266 (+ 1504us) server_negotiation.cc:407] Connection header received
0620 18:34:18.568553 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:18.568563 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:18.570327 (+ 1764us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:18.570486 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:18.571308 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.573934 (+ 2626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.576421 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.578936 (+ 2515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.579010 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.579147 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:18.579595 (+ 448us) server_negotiation.cc:299] Negotiation successful
0620 18:34:18.579757 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":252,"threads_started":1}
I20260620 18:34:18.582549 22524 tablet_service.cc:1467] Processing CreateTablet for tablet 7b7b9338f17d413fbbba3081be9c9374 (DEFAULT_TABLE table=test-table [id=858db983b3074f1599a09baab03be4b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:18.584195 22524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b7b9338f17d413fbbba3081be9c9374. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:18.587143 22247 tablet_service.cc:1467] Processing CreateTablet for tablet 7b7b9338f17d413fbbba3081be9c9374 (DEFAULT_TABLE table=test-table [id=858db983b3074f1599a09baab03be4b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:18.589111 22247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b7b9338f17d413fbbba3081be9c9374. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:18.590418 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.568220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:34939 (local address 127.0.0.1:43894)
0620 18:34:18.577468 (+ 9248us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:18.577487 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:18.577571 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:18.582791 (+ 5220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:18.582795 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:18.582820 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:18.583020 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.583032 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.586029 (+ 2997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.586034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.587852 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.587868 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.589644 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.589652 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.589851 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.589884 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:34:18.589987 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 18:34:18.590129 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9140,"thread_start_us":15039,"threads_started":1}
I20260620 18:34:18.590714 22606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.568509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43894 (local address 127.10.131.2:34939)
0620 18:34:18.581358 (+ 12849us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:18.581371 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:18.581409 (+ 38us) server_negotiation.cc:407] Connection header received
0620 18:34:18.581528 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:18.581539 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:18.581816 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:18.583201 (+ 1385us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:18.583426 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.585845 (+ 2419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.588035 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.589461 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.589616 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.589806 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:18.590036 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 18:34:18.590199 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12660,"thread_start_us":12518,"threads_started":1}
I20260620 18:34:18.594735 22390 tablet_service.cc:1467] Processing CreateTablet for tablet 7b7b9338f17d413fbbba3081be9c9374 (DEFAULT_TABLE table=test-table [id=858db983b3074f1599a09baab03be4b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:34:18.596236 22390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b7b9338f17d413fbbba3081be9c9374. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:34:18.597121 22607 tablet_bootstrap.cc:492] T 7b7b9338f17d413fbbba3081be9c9374 P 589b9774bfdd47bf850f2da794ce8296: Bootstrap starting.
I20260620 18:34:18.600111 22607 tablet_bootstrap.cc:654] T 7b7b9338f17d413fbbba3081be9c9374 P 589b9774bfdd47bf850f2da794ce8296: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:18.601518 22607 log.cc:824] T 7b7b9338f17d413fbbba3081be9c9374 P 589b9774bfdd47bf850f2da794ce8296: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:18.603873 22609 tablet_bootstrap.cc:492] T 7b7b9338f17d413fbbba3081be9c9374 P 01c58f5ea6e94355ab61a3b194f01307: Bootstrap starting.
I20260620 18:34:18.605070 22607 tablet_bootstrap.cc:492] T 7b7b9338f17d413fbbba3081be9c9374 P 589b9774bfdd47bf850f2da794ce8296: No bootstrap required, opened a new log
I20260620 18:34:18.605370 22607 ts_tablet_manager.cc:1397] T 7b7b9338f17d413fbbba3081be9c9374 P 589b9774bfdd47bf850f2da794ce8296: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260620 18:34:18.608014 22609 tablet_bootstrap.cc:654] T 7b7b9338f17d413fbbba3081be9c9374 P 01c58f5ea6e94355ab61a3b194f01307: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:18.609190 22610 tablet_bootstrap.cc:492] T 7b7b9338f17d413fbbba3081be9c9374 P 9a720b4005b041f4803fe0fefff4650b: Bootstrap starting.
I20260620 18:34:18.609208 22609 log.cc:824] T 7b7b9338f17d413fbbba3081be9c9374 P 01c58f5ea6e94355ab61a3b194f01307: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:18.611919 22610 tablet_bootstrap.cc:654] T 7b7b9338f17d413fbbba3081be9c9374 P 9a720b4005b041f4803fe0fefff4650b: Neither blocks nor log segments found. Creating new log.
I20260620 18:34:18.613112 22610 log.cc:824] T 7b7b9338f17d413fbbba3081be9c9374 P 9a720b4005b041f4803fe0fefff4650b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:34:18.615844 22609 tablet_bootstrap.cc:492] T 7b7b9338f17d413fbbba3081be9c9374 P 01c58f5ea6e94355ab61a3b194f01307: No bootstrap required, opened a new log
I20260620 18:34:18.616101 22609 ts_tablet_manager.cc:1397] T 7b7b9338f17d413fbbba3081be9c9374 P 01c58f5ea6e94355ab61a3b194f01307: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.004s
I20260620 18:34:18.617709 22607 raft_consensus.cc:348] T 7b7b9338f17d413fbbba3081be9c9374 P 589b9774bfdd47bf850f2da794ce8296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589b9774bfdd47bf850f2da794ce8296" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37057 } } peers { permanent_uuid: "01c58f5ea6e94355ab61a3b194f01307" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 36351 } } peers { permanent_uuid: "9a720b4005b041f4803fe0fefff4650b" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 34939 } }
I20260620 18:34:18.618152 22607 raft_consensus.cc:374] T 7b7b9338f17d413fbbba3081be9c9374 P 589b9774bfdd47bf850f2da794ce8296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:18.618325 22607 raft_consensus.cc:729] T 7b7b9338f17d413fbbba3081be9c9374 P 589b9774bfdd47bf850f2da794ce8296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 589b9774bfdd47bf850f2da794ce8296, State: Initialized, Role: FOLLOWER
I20260620 18:34:18.618979 22607 consensus_queue.cc:260] T 7b7b9338f17d413fbbba3081be9c9374 P 589b9774bfdd47bf850f2da794ce8296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589b9774bfdd47bf850f2da794ce8296" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37057 } } peers { permanent_uuid: "01c58f5ea6e94355ab61a3b194f01307" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 36351 } } peers { permanent_uuid: "9a720b4005b041f4803fe0fefff4650b" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 34939 } }
I20260620 18:34:18.619397 22610 tablet_bootstrap.cc:492] T 7b7b9338f17d413fbbba3081be9c9374 P 9a720b4005b041f4803fe0fefff4650b: No bootstrap required, opened a new log
I20260620 18:34:18.619707 22610 ts_tablet_manager.cc:1397] T 7b7b9338f17d413fbbba3081be9c9374 P 9a720b4005b041f4803fe0fefff4650b: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20260620 18:34:18.620216 22589 heartbeater.cc:503] Master 127.10.131.62:43609 was elected leader, sending a full tablet report...
I20260620 18:34:18.620698 22607 ts_tablet_manager.cc:1428] T 7b7b9338f17d413fbbba3081be9c9374 P 589b9774bfdd47bf850f2da794ce8296: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
I20260620 18:34:18.626235 22609 raft_consensus.cc:348] T 7b7b9338f17d413fbbba3081be9c9374 P 01c58f5ea6e94355ab61a3b194f01307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589b9774bfdd47bf850f2da794ce8296" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37057 } } peers { permanent_uuid: "01c58f5ea6e94355ab61a3b194f01307" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 36351 } } peers { permanent_uuid: "9a720b4005b041f4803fe0fefff4650b" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 34939 } }
I20260620 18:34:18.626660 22609 raft_consensus.cc:374] T 7b7b9338f17d413fbbba3081be9c9374 P 01c58f5ea6e94355ab61a3b194f01307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:18.626763 22609 raft_consensus.cc:729] T 7b7b9338f17d413fbbba3081be9c9374 P 01c58f5ea6e94355ab61a3b194f01307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01c58f5ea6e94355ab61a3b194f01307, State: Initialized, Role: FOLLOWER
I20260620 18:34:18.627378 22609 consensus_queue.cc:260] T 7b7b9338f17d413fbbba3081be9c9374 P 01c58f5ea6e94355ab61a3b194f01307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589b9774bfdd47bf850f2da794ce8296" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37057 } } peers { permanent_uuid: "01c58f5ea6e94355ab61a3b194f01307" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 36351 } } peers { permanent_uuid: "9a720b4005b041f4803fe0fefff4650b" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 34939 } }
I20260620 18:34:18.629083 22609 ts_tablet_manager.cc:1428] T 7b7b9338f17d413fbbba3081be9c9374 P 01c58f5ea6e94355ab61a3b194f01307: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260620 18:34:18.631484 22610 raft_consensus.cc:348] T 7b7b9338f17d413fbbba3081be9c9374 P 9a720b4005b041f4803fe0fefff4650b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589b9774bfdd47bf850f2da794ce8296" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37057 } } peers { permanent_uuid: "01c58f5ea6e94355ab61a3b194f01307" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 36351 } } peers { permanent_uuid: "9a720b4005b041f4803fe0fefff4650b" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 34939 } }
I20260620 18:34:18.631866 22610 raft_consensus.cc:374] T 7b7b9338f17d413fbbba3081be9c9374 P 9a720b4005b041f4803fe0fefff4650b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:34:18.631970 22610 raft_consensus.cc:729] T 7b7b9338f17d413fbbba3081be9c9374 P 9a720b4005b041f4803fe0fefff4650b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a720b4005b041f4803fe0fefff4650b, State: Initialized, Role: FOLLOWER
I20260620 18:34:18.632567 22610 consensus_queue.cc:260] T 7b7b9338f17d413fbbba3081be9c9374 P 9a720b4005b041f4803fe0fefff4650b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589b9774bfdd47bf850f2da794ce8296" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37057 } } peers { permanent_uuid: "01c58f5ea6e94355ab61a3b194f01307" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 36351 } } peers { permanent_uuid: "9a720b4005b041f4803fe0fefff4650b" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 34939 } }
I20260620 18:34:18.633471 22455 heartbeater.cc:503] Master 127.10.131.62:43609 was elected leader, sending a full tablet report...
I20260620 18:34:18.634056 22610 ts_tablet_manager.cc:1428] T 7b7b9338f17d413fbbba3081be9c9374 P 9a720b4005b041f4803fe0fefff4650b: Time spent starting tablet: real 0.014s user 0.006s sys 0.006s
W20260620 18:34:18.638495 22313 tablet.cc:2369] T 7b7b9338f17d413fbbba3081be9c9374 P 01c58f5ea6e94355ab61a3b194f01307: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:34:18.724115 22456 tablet.cc:2369] T 7b7b9338f17d413fbbba3081be9c9374 P 9a720b4005b041f4803fe0fefff4650b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:34:18.749406 22590 tablet.cc:2369] T 7b7b9338f17d413fbbba3081be9c9374 P 589b9774bfdd47bf850f2da794ce8296: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:34:18.925625 22615 raft_consensus.cc:482] T 7b7b9338f17d413fbbba3081be9c9374 P 9a720b4005b041f4803fe0fefff4650b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:34:18.925935 22615 raft_consensus.cc:504] T 7b7b9338f17d413fbbba3081be9c9374 P 9a720b4005b041f4803fe0fefff4650b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589b9774bfdd47bf850f2da794ce8296" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37057 } } peers { permanent_uuid: "01c58f5ea6e94355ab61a3b194f01307" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 36351 } } peers { permanent_uuid: "9a720b4005b041f4803fe0fefff4650b" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 34939 } }
I20260620 18:34:18.927735 22615 leader_election.cc:288] T 7b7b9338f17d413fbbba3081be9c9374 P 9a720b4005b041f4803fe0fefff4650b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 589b9774bfdd47bf850f2da794ce8296 (127.10.131.3:37057), 01c58f5ea6e94355ab61a3b194f01307 (127.10.131.1:36351)
I20260620 18:34:18.935286 22604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.928310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:43145 (local address 127.10.131.3:37057)
0620 18:34:18.928517 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:18.928525 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:18.929748 (+ 1223us) server_negotiation.cc:407] Connection header received
0620 18:34:18.930056 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:18.930064 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:18.930148 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:18.930253 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:18.930913 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.931701 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.933524 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.934318 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.934437 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.934545 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:18.934916 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 18:34:18.935038 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:34:18.935310 22602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.928329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:49417 (local address 127.10.131.1:36351)
0620 18:34:18.928873 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:18.928881 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:18.929949 (+ 1068us) server_negotiation.cc:407] Connection header received
0620 18:34:18.930134 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:18.930139 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:18.930218 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:18.930306 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:34:18.931140 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.931887 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.933606 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.934356 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.934552 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.934623 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:18.935030 (+ 407us) server_negotiation.cc:299] Negotiation successful
0620 18:34:18.935133 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 18:34:18.935410 22616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.928154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:37057 (local address 127.10.131.2:43145)
0620 18:34:18.929617 (+ 1463us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:18.929642 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:18.929875 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:18.930268 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:18.930275 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:18.930301 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:18.930768 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.930778 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.931832 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.931839 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.933381 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.933393 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.934545 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.934552 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.934763 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.934786 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 18:34:18.934890 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 18:34:18.935012 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1286,"mutex_wait_us":9,"thread_start_us":1043,"threads_started":1}
I20260620 18:34:18.936059 22617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:18.928292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:36351 (local address 127.10.131.2:49417)
0620 18:34:18.929839 (+ 1547us) negotiation.cc:107] Waiting for socket to connect
0620 18:34:18.929857 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:34:18.930007 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:34:18.930461 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:34:18.930466 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:34:18.930524 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:34:18.930984 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.930997 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.932018 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.932022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.933441 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:34:18.933473 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:34:18.934601 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:34:18.934605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:34:18.934844 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:18.934877 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:34:18.935546 (+ 669us) client_negotiation.cc:240] Negotiation successful
0620 18:34:18.935687 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1412,"mutex_wait_us":40,"thread_start_us":1228,"threads_started":1}
I20260620 18:34:18.936491 22544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b7b9338f17d413fbbba3081be9c9374" candidate_uuid: "9a720b4005b041f4803fe0fefff4650b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "589b9774bfdd47bf850f2da794ce8296" is_pre_election: true
I20260620 18:34:18.937086 22544 raft_consensus.cc:2454] T 7b7b9338f17d413fbbba3081be9c9374 P 589b9774bfdd47bf850f2da794ce8296 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a720b4005b041f4803fe0fefff4650b in term 0.
I20260620 18:34:18.937024 22267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b7b9338f17d413fbbba3081be9c9374" candidate_uuid: "9a720b4005b041f4803fe0fefff4650b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01c58f5ea6e94355ab61a3b194f01307" is_pre_election: true
I20260620 18:34:18.938055 22267 raft_consensus.cc:2454] T 7b7b9338f17d413fbbba3081be9c9374 P 01c58f5ea6e94355ab61a3b194f01307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a720b4005b041f4803fe0fefff4650b in term 0.
I20260620 18:34:18.938156 22346 leader_election.cc:302] T 7b7b9338f17d413fbbba3081be9c9374 P 9a720b4005b041f4803fe0fefff4650b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 589b9774bfdd47bf850f2da794ce8296, 9a720b4005b041f4803fe0fefff4650b; no voters:
I20260620 18:34:18.938565 22615 raft_consensus.cc:2781] T 7b7b9338f17d413fbbba3081be9c9374 P 9a720b4005b041f4803fe0fefff4650b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:34:18.938699 22615 raft_consensus.cc:482] T 7b7b9338f17d413fbbba3081be9c9374 P 9a720b4005b041f4803fe0fefff4650b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:34:18.938833 22615 raft_consensus.cc:3037] T 7b7b9338f17d413fbbba3081be9c9374 P 9a720b4005b041f4803fe0fefff4650b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:18.941066 22615 raft_consensus.cc:504] T 7b7b9338f17d413fbbba3081be9c9374 P 9a720b4005b041f4803fe0fefff4650b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589b9774bfdd47bf850f2da794ce8296" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37057 } } peers { permanent_uuid: "01c58f5ea6e94355ab61a3b194f01307" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 36351 } } peers { permanent_uuid: "9a720b4005b041f4803fe0fefff4650b" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 34939 } }
I20260620 18:34:18.941716 22615 leader_election.cc:288] T 7b7b9338f17d413fbbba3081be9c9374 P 9a720b4005b041f4803fe0fefff4650b [CANDIDATE]: Term 1 election: Requested vote from peers 589b9774bfdd47bf850f2da794ce8296 (127.10.131.3:37057), 01c58f5ea6e94355ab61a3b194f01307 (127.10.131.1:36351)
I20260620 18:34:18.942080 22544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b7b9338f17d413fbbba3081be9c9374" candidate_uuid: "9a720b4005b041f4803fe0fefff4650b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "589b9774bfdd47bf850f2da794ce8296"
I20260620 18:34:18.942179 22267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b7b9338f17d413fbbba3081be9c9374" candidate_uuid: "9a720b4005b041f4803fe0fefff4650b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01c58f5ea6e94355ab61a3b194f01307"
I20260620 18:34:18.942291 22544 raft_consensus.cc:3037] T 7b7b9338f17d413fbbba3081be9c9374 P 589b9774bfdd47bf850f2da794ce8296 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:18.942376 22267 raft_consensus.cc:3037] T 7b7b9338f17d413fbbba3081be9c9374 P 01c58f5ea6e94355ab61a3b194f01307 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:34:18.945461 22267 raft_consensus.cc:2454] T 7b7b9338f17d413fbbba3081be9c9374 P 01c58f5ea6e94355ab61a3b194f01307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a720b4005b041f4803fe0fefff4650b in term 1.
I20260620 18:34:18.945539 22544 raft_consensus.cc:2454] T 7b7b9338f17d413fbbba3081be9c9374 P 589b9774bfdd47bf850f2da794ce8296 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a720b4005b041f4803fe0fefff4650b in term 1.
I20260620 18:34:18.945880 22344 leader_election.cc:302] T 7b7b9338f17d413fbbba3081be9c9374 P 9a720b4005b041f4803fe0fefff4650b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01c58f5ea6e94355ab61a3b194f01307, 9a720b4005b041f4803fe0fefff4650b; no voters:
I20260620 18:34:18.946231 22615 raft_consensus.cc:2781] T 7b7b9338f17d413fbbba3081be9c9374 P 9a720b4005b041f4803fe0fefff4650b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:34:18.946699 22615 raft_consensus.cc:686] T 7b7b9338f17d413fbbba3081be9c9374 P 9a720b4005b041f4803fe0fefff4650b [term 1 LEADER]: Becoming Leader. State: Replica: 9a720b4005b041f4803fe0fefff4650b, State: Running, Role: LEADER
I20260620 18:34:18.947237 22615 consensus_queue.cc:237] T 7b7b9338f17d413fbbba3081be9c9374 P 9a720b4005b041f4803fe0fefff4650b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589b9774bfdd47bf850f2da794ce8296" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37057 } } peers { permanent_uuid: "01c58f5ea6e94355ab61a3b194f01307" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 36351 } } peers { permanent_uuid: "9a720b4005b041f4803fe0fefff4650b" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 34939 } }
I20260620 18:34:18.951998 22099 catalog_manager.cc:5495] T 7b7b9338f17d413fbbba3081be9c9374 P 9a720b4005b041f4803fe0fefff4650b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a720b4005b041f4803fe0fefff4650b (127.10.131.2). New cstate: current_term: 1 leader_uuid: "9a720b4005b041f4803fe0fefff4650b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589b9774bfdd47bf850f2da794ce8296" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01c58f5ea6e94355ab61a3b194f01307" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 36351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a720b4005b041f4803fe0fefff4650b" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 34939 } health_report { overall_health: HEALTHY } } }
I20260620 18:34:19.030570 22606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:19.025152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43908 (local address 127.10.131.2:34939)
0620 18:34:19.025329 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:19.025335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:19.025420 (+ 85us) server_negotiation.cc:407] Connection header received
0620 18:34:19.025597 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:19.025600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:19.025695 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:19.025852 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:34:19.026136 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.026862 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.028379 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.028787 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.029061 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:19.029548 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:34:19.030086 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:34:19.030193 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:19.030282 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 18:34:19.030358 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 18:34:19.038839 22267 raft_consensus.cc:1261] T 7b7b9338f17d413fbbba3081be9c9374 P 01c58f5ea6e94355ab61a3b194f01307 [term 1 FOLLOWER]: Refusing update from remote peer 9a720b4005b041f4803fe0fefff4650b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:19.038870 22544 raft_consensus.cc:1261] T 7b7b9338f17d413fbbba3081be9c9374 P 589b9774bfdd47bf850f2da794ce8296 [term 1 FOLLOWER]: Refusing update from remote peer 9a720b4005b041f4803fe0fefff4650b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:34:19.039808 22618 consensus_queue.cc:1035] T 7b7b9338f17d413fbbba3081be9c9374 P 9a720b4005b041f4803fe0fefff4650b [LEADER]: Connected to new peer: Peer: permanent_uuid: "01c58f5ea6e94355ab61a3b194f01307" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260620 18:34:19.040619 22615 consensus_queue.cc:1035] T 7b7b9338f17d413fbbba3081be9c9374 P 9a720b4005b041f4803fe0fefff4650b [LEADER]: Connected to new peer: Peer: permanent_uuid: "589b9774bfdd47bf850f2da794ce8296" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:34:19.052584 22622 mvcc.cc:205] Tried to move back new op lower bound from 7298991960212852736 to 7298991959852093440. Current Snapshot: MvccSnapshot[applied={T|T < 7298991960212852736}]
I20260620 18:34:19.065948 22099 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:19.066206 22099 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:34:19.069415 22099 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9bd1dafd754c42689d70bbb7227f700a: Sending DeleteTablet for 3 replicas of tablet 7b7b9338f17d413fbbba3081be9c9374
I20260620 18:34:19.070511 22524 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b7b9338f17d413fbbba3081be9c9374 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:19 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:34640
I20260620 18:34:19.070570 22390 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b7b9338f17d413fbbba3081be9c9374 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:19 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:43894
I20260620 18:34:19.070566 22247 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b7b9338f17d413fbbba3081be9c9374 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:34:19 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:39250
I20260620 18:34:19.073505 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 22179
I20260620 18:34:19.083514 22633 tablet_replica.cc:330] T 7b7b9338f17d413fbbba3081be9c9374 P 9a720b4005b041f4803fe0fefff4650b: stopping tablet replica
I20260620 18:34:19.084115 22633 raft_consensus.cc:2229] T 7b7b9338f17d413fbbba3081be9c9374 P 9a720b4005b041f4803fe0fefff4650b [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:34:19.084991 22633 raft_consensus.cc:2258] T 7b7b9338f17d413fbbba3081be9c9374 P 9a720b4005b041f4803fe0fefff4650b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:19.085690 22631 tablet_replica.cc:330] T 7b7b9338f17d413fbbba3081be9c9374 P 589b9774bfdd47bf850f2da794ce8296: stopping tablet replica
I20260620 18:34:19.086194 22631 raft_consensus.cc:2229] T 7b7b9338f17d413fbbba3081be9c9374 P 589b9774bfdd47bf850f2da794ce8296 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:19.086807 22631 raft_consensus.cc:2258] T 7b7b9338f17d413fbbba3081be9c9374 P 589b9774bfdd47bf850f2da794ce8296 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:19.089223 22631 ts_tablet_manager.cc:1905] T 7b7b9338f17d413fbbba3081be9c9374 P 589b9774bfdd47bf850f2da794ce8296: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:19.093029 22633 ts_tablet_manager.cc:1905] T 7b7b9338f17d413fbbba3081be9c9374 P 9a720b4005b041f4803fe0fefff4650b: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 18:34:19 dist-test-slave-cbn3 krb5kdc[21623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.1@KRBTEST.COM
I20260620 18:34:19.101433 22632 tablet_replica.cc:330] T 7b7b9338f17d413fbbba3081be9c9374 P 01c58f5ea6e94355ab61a3b194f01307: stopping tablet replica
I20260620 18:34:19.101882 22632 raft_consensus.cc:2229] T 7b7b9338f17d413fbbba3081be9c9374 P 01c58f5ea6e94355ab61a3b194f01307 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:34:19.102465 22632 raft_consensus.cc:2258] T 7b7b9338f17d413fbbba3081be9c9374 P 01c58f5ea6e94355ab61a3b194f01307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:34:19.104975 22632 ts_tablet_manager.cc:1905] T 7b7b9338f17d413fbbba3081be9c9374 P 01c58f5ea6e94355ab61a3b194f01307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:34:19.107213 22631 ts_tablet_manager.cc:1918] T 7b7b9338f17d413fbbba3081be9c9374 P 589b9774bfdd47bf850f2da794ce8296: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:34:19.107427 22631 log.cc:1196] T 7b7b9338f17d413fbbba3081be9c9374 P 589b9774bfdd47bf850f2da794ce8296: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-2/wal/wals/7b7b9338f17d413fbbba3081be9c9374
I20260620 18:34:19.108412 22631 ts_tablet_manager.cc:1939] T 7b7b9338f17d413fbbba3081be9c9374 P 589b9774bfdd47bf850f2da794ce8296: Deleting consensus metadata
I20260620 18:34:19.109090 22602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:19.074939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39264 (local address 127.10.131.1:36351)
0620 18:34:19.075114 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:19.075120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:19.088335 (+ 13215us) server_negotiation.cc:407] Connection header received
0620 18:34:19.088818 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:19.088824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:19.088934 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:19.089022 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:19.090992 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.092152 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.094203 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.094897 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.094995 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:19.100717 (+ 5722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:19.100748 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:19.100767 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:19.100813 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:19.105202 (+ 4389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:19.106161 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:19.106170 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:19.106174 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:19.106336 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:19.106834 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:19.106839 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:19.106844 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:19.107308 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:19.107703 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:19.108294 (+ 591us) server_negotiation.cc:299] Negotiation successful
0620 18:34:19.108600 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:34:19.110402 22086 catalog_manager.cc:4841] TS 589b9774bfdd47bf850f2da794ce8296 (127.10.131.3:37057): tablet 7b7b9338f17d413fbbba3081be9c9374 (table test-table [id=858db983b3074f1599a09baab03be4b3]) successfully deleted
I20260620 18:34:19.111510 22307 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:34:19.112017 22633 ts_tablet_manager.cc:1918] T 7b7b9338f17d413fbbba3081be9c9374 P 9a720b4005b041f4803fe0fefff4650b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:34:19.112201 22633 log.cc:1196] T 7b7b9338f17d413fbbba3081be9c9374 P 9a720b4005b041f4803fe0fefff4650b: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-1/wal/wals/7b7b9338f17d413fbbba3081be9c9374
I20260620 18:34:19.112757 22633 ts_tablet_manager.cc:1939] T 7b7b9338f17d413fbbba3081be9c9374 P 9a720b4005b041f4803fe0fefff4650b: Deleting consensus metadata
I20260620 18:34:19.115062 22084 catalog_manager.cc:4841] TS 9a720b4005b041f4803fe0fefff4650b (127.10.131.2:34939): tablet 7b7b9338f17d413fbbba3081be9c9374 (table test-table [id=858db983b3074f1599a09baab03be4b3]) successfully deleted
I20260620 18:34:19.199925 22632 ts_tablet_manager.cc:1918] T 7b7b9338f17d413fbbba3081be9c9374 P 01c58f5ea6e94355ab61a3b194f01307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:34:19.200222 22632 log.cc:1196] T 7b7b9338f17d413fbbba3081be9c9374 P 01c58f5ea6e94355ab61a3b194f01307: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781980314095189-10764-0/minicluster-data/ts-0/wal/wals/7b7b9338f17d413fbbba3081be9c9374
I20260620 18:34:19.200773 22632 ts_tablet_manager.cc:1939] T 7b7b9338f17d413fbbba3081be9c9374 P 01c58f5ea6e94355ab61a3b194f01307: Deleting consensus metadata
I20260620 18:34:19.202569 22084 catalog_manager.cc:4841] TS 01c58f5ea6e94355ab61a3b194f01307 (127.10.131.1:36351): tablet 7b7b9338f17d413fbbba3081be9c9374 (table test-table [id=858db983b3074f1599a09baab03be4b3]) successfully deleted
I20260620 18:34:19.205534 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 22179
I20260620 18:34:19.223091 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 22318
Jun 20 18:34:19 dist-test-slave-cbn3 krb5kdc[21623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.2@KRBTEST.COM
I20260620 18:34:19.240588 22606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:19.223898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43912 (local address 127.10.131.2:34939)
0620 18:34:19.224118 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:19.224126 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:19.224146 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:34:19.224398 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:19.224402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:19.224466 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:19.224545 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:19.226221 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.227014 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.228917 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.229336 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.229435 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:19.234717 (+ 5282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:19.234733 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:19.234749 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:19.234785 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:19.237962 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:19.238727 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:19.238734 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:19.238736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:19.238882 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:19.239245 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:19.239248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:19.239250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:19.239564 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:19.239793 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:19.240139 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 18:34:19.240298 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 18:34:19.241454 22450 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:34:19.352170 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 22318
I20260620 18:34:19.369343 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 22461
Jun 20 18:34:19 dist-test-slave-cbn3 krb5kdc[21623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.3@KRBTEST.COM
I20260620 18:34:19.384716 22604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:34:19.370043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34652 (local address 127.10.131.3:37057)
0620 18:34:19.370218 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 18:34:19.370225 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:34:19.370280 (+ 55us) server_negotiation.cc:407] Connection header received
0620 18:34:19.370498 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:34:19.370502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:34:19.370571 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:34:19.370674 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:34:19.372011 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.372718 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.374037 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:34:19.374437 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:34:19.374582 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:34:19.378942 (+ 4360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:34:19.378958 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:34:19.378975 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:34:19.379007 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:34:19.381859 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:19.382580 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:19.382589 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:19.382591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:19.382743 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:34:19.383133 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:34:19.383137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:34:19.383140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:34:19.383499 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:34:19.383829 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0620 18:34:19.384145 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 18:34:19.384368 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:34:19.385452 22584 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:34:19.473712 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 22461
I20260620 18:34:19.490016 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 22067
I20260620 18:34:19.491179 22129 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:34:19.600171 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 22067
I20260620 18:34:19.625216 10764 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 18:34:19.959970 10764 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 18:34:19.960216 10764 mini_ranger.cc:66] Stopping Ranger...
I20260620 18:34:20.305491 10764 mini_ranger.cc:68] Stopped Ranger
2026-06-20 18:34:20.305 UTC [21653] LOG: received smart shutdown request
2026-06-20 18:34:20.308 UTC [21653] LOG: background worker "logical replication launcher" (PID 21662) exited with exit code 1
2026-06-20 18:34:20.309 UTC [21657] LOG: shutting down
2026-06-20 18:34:20.438 UTC [21653] LOG: database system is shut down
2026-06-20T18:34:20Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49054 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:34:20 dist-test-slave-cbn3 krb5kdc[22647](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:34:20 dist-test-slave-cbn3 krb5kdc[22647](info): set up 2 sockets
Jun 20 18:34:20 dist-test-slave-cbn3 krb5kdc[22647](info): commencing operation
krb5kdc: starting...
W20260620 18:34:21.682606 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:34:21 dist-test-slave-cbn3 krb5kdc[22647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:34:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:34:21Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.10.131.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.10.131.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.10.131.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/rangeradmin_127.10.131.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.10.131.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/rangeradmin_127.10.131.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.10.131.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.10.131.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.10.131.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/rangerlookup_127.10.131.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.10.131.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/rangerlookup_127.10.131.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.131.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.131.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.131.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/HTTP_127.10.131.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.131.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/HTTP_127.10.131.20@KRBTEST.COM.keytab.
I20260620 18:34:22.003964 10764 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task0dhafo/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/postgres -l logfile start
2026-06-20 18:34:24.354 UTC [22677] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 18:34:24.354 UTC [22677] LOG: listening on IPv4 address "127.10.131.20", port 48583
2026-06-20 18:34:24.357 UTC [22677] LOG: listening on Unix socket "/tmp/.s.PGSQL.48583"
2026-06-20 18:34:24.378 UTC [22680] LOG: database system was shut down at 2026-06-20 18:34:23 UTC
2026-06-20 18:34:24.383 UTC [22677] LOG: database system is ready to accept connections
I20260620 18:34:25.357187 10764 mini_postgres.cc:96] Postgres bound to 48583
2026-06-20 18:34:25.374 UTC [22688] FATAL: database "slave" does not exist
127.10.131.20:48583 - accepting connections
I20260620 18:34:25.374974 10764 mini_ranger.cc:161] Starting Ranger...
I20260620 18:34:25.402207 10764 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 18:34:25.843284 10764 mini_ranger.cc:87] Created ranger Postgres database
I20260620 18:34:25.843380 10764 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ranger-admin
2026-06-20 18:34:26,132 [I] DB FLAVOR :POSTGRES
2026-06-20 18:34:26,132 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 18:34:26,132 [I] Checking connection..
2026-06-20 18:34:26,132 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:48583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 18:34:26,390 [I] Checking connection passed.
2026-06-20 18:34:26,390 [I] --------- Verifying version history table ---------
2026-06-20 18:34:26,390 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:48583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:34:26,620 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 18:34:26,621 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:48583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:34:26,865 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 18:34:26,865 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 18:34:26,865 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:48583/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 18:34:27.121 UTC [22752] WARNING: there is no transaction in progress
2026-06-20 18:34:27,127 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 18:34:27,128 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:48583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:34:27,355 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 18:34:27,356 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 18:34:27,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:48583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 18:34:27,582 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:48583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 18:34:27,847 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:48583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-cbn3.c.gcp-upstream.internal','N') ;"
2026-06-20 18:34:28,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:48583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 18:34:28,327 [I] Table x_portal_user does not exist in database ranger
2026-06-20 18:34:28,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:48583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 18:34:28,568 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 18:34:28,569 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:48583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 18:34:28,830 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:48583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 18:34:29,069 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 18:34:29,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:48583/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 18:34:29.995 UTC [22860] WARNING: there is no transaction in progress
2026-06-20 18:34:30.010 UTC [22860] WARNING: there is no transaction in progress
2026-06-20 18:34:30.026 UTC [22860] WARNING: there is no transaction in progress
2026-06-20 18:34:30.041 UTC [22860] WARNING: there is no transaction in progress
2026-06-20 18:34:30.061 UTC [22860] WARNING: there is no transaction in progress
2026-06-20 18:34:30.078 UTC [22860] WARNING: there is no transaction in progress
2026-06-20 18:34:30.130 UTC [22860] WARNING: there is no transaction in progress
2026-06-20 18:34:30.136 UTC [22860] WARNING: there is no transaction in progress
2026-06-20 18:34:30.149 UTC [22860] WARNING: there is no transaction in progress
2026-06-20 18:34:30.158 UTC [22860] WARNING: there is no transaction in progress
2026-06-20 18:34:30.168 UTC [22860] WARNING: there is no transaction in progress
2026-06-20 18:34:30.181 UTC [22860] WARNING: there is no transaction in progress
2026-06-20 18:34:30.189 UTC [22860] WARNING: there is no transaction in progress
2026-06-20 18:34:30.194 UTC [22860] WARNING: there is no transaction in progress
2026-06-20 18:34:30.200 UTC [22860] WARNING: there is no transaction in progress
2026-06-20 18:34:30.206 UTC [22860] WARNING: there is no transaction in progress
2026-06-20 18:34:30,767 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 18:34:30,767 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:48583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 18:34:31,005 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 18:34:31,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:48583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 18:34:31,241 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 18:34:31,241 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:48583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 18:34:31,494 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 18:34:31,495 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:48583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 18:34:31,767 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:48583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 18:34:32,026 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:48583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-cbn3.c.gcp-upstream.internal';"
2026-06-20 18:34:32,258 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 18:34:32,259 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:48583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 18:34:32,517 [I] DB_PATCHES have already been applied
I20260620 18:34:32.523041 10764 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ranger-admin:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task0dhafo/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 18:34:32.523132 10764 mini_ranger.cc:193] Using host: 127.10.131.20
I20260620 18:34:32.537492 10764 mini_ranger.cc:227] Ranger admin URL: http://127.10.131.20:46245
Jun 20, 2026 6:34:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:34:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:34:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 6:34:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 6:34:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 6:34:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 6:34:33 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 18:34:33.704102 10764 mini_ranger.cc:160] Time spent starting Ranger: real 8.329s user 0.002s sys 0.036s
Jun 20, 2026 6:34:52 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 6:34:52 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 20, 2026 6:34:52 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 6:34:53 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 6:34:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 6:34:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 6:34:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 6:34:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 6:34:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 6:34:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 6:34:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 6:34:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 6:34:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 6:34:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 6:34:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 6:34:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 6:34:53 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 6:34:53 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 18:34:54.499307 10764 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.10.131.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.10.131.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.10.131.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/rangerkms_127.10.131.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.10.131.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/rangerkms_127.10.131.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.131.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.10.131.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.131.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/HTTP_127.10.131.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.131.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/HTTP_127.10.131.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 18:34:54 dist-test-slave-cbn3 krb5kdc[22647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980494, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 18:34:54.661005 10764 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 18:34:54.684113 10764 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 18:34:55.409737 10764 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 18:34:55.409855 10764 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ranger-kms
I20260620 18:34:55.409937 10764 mini_ranger_kms.cc:227] Using postgres at 127.10.131.20:48583
2026-06-20 18:34:55,756 [I] DB FLAVOR :POSTGRES
2026-06-20 18:34:55,757 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 18:34:55,757 [I] Checking connection
2026-06-20 18:34:55,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:48583/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 18:34:55,996 [I] connection success
2026-06-20 18:34:55,997 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 18:34:55,997 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 18:34:55,997 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:48583/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 18:34:56,228 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 18:34:56,229 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 18:34:56,229 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 18:34:56,229 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.131.20:48583/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 18:34:56,491 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 18:34:56.937394 10764 mini_ranger_kms.cc:310] Created kms service
I20260620 18:34:57.118266 10764 mini_ranger_kms.cc:326] Created kudu user
I20260620 18:34:57.161738 10764 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 18:34:57 dist-test-slave-cbn3 krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980494, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.10.131.20@KRBTEST.COM
I20260620 18:34:57.554050 10764 mini_ranger_kms.cc:384] Added ranger policy
I20260620 18:34:57.554369 10764 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ranger-kms:/tmp/dist-test-task0dhafo/build/asan/bin/postgresql.jar:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task0dhafo/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 18:34:57.554419 10764 mini_ranger_kms.cc:240] Using host: 127.10.131.20
I20260620 18:34:57.572985 10764 mini_ranger_kms.cc:276] Ranger KMS PID: 23040
I20260620 18:34:57.573112 10764 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.10.131.20:60667
Jun 20, 2026 6:34:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:34:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:34:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 6:34:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 6:34:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 6:34:59 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 18:34:59.840096 10764 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.179s user 0.062s sys 0.300s
I20260620 18:34:59.840333 10764 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 18:34:59.840513 10764 mini_ranger_kms.cc:401] 127.10.131.20:60667/kms/v1/keys
Jun 20 18:34:59 dist-test-slave-cbn3 krb5kdc[22647](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781980499, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.131.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 18:35:00 dist-test-slave-cbn3 krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980499, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.131.20@KRBTEST.COM for HTTP/127.10.131.20@KRBTEST.COM
Jun 20 18:35:00 dist-test-slave-cbn3 krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980499, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.131.20@KRBTEST.COM for HTTP/127.10.131.20@KRBTEST.COM
Jun 20 18:35:02 dist-test-slave-cbn3 krb5kdc[22647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.10.131.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
I20260620 18:35:02.982609 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:32927
--webserver_interface=127.10.131.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:44799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:32927
--ranger_config_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.131.20:60667/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:03.319195 23091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:03.319548 23091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:03.319764 23091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:03.327322 23091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:35:03.327472 23091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:03.327550 23091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:03.327630 23091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:35:03.327696 23091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:35:03.337430 23091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:44799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/master-0/wal
--ranger_kms_url=127.10.131.20:60667/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:32927
--ranger_config_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:32927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:35:03.338737 23091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:03.340040 23091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:03.352319 23096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:03.352406 23097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:03.352507 23091 server_base.cc:1034] running on GCE node
W20260620 18:35:03.352406 23099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:03.353907 23091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:03.355268 23091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:03.356488 23091 hybrid_clock.cc:648] HybridClock initialized: now 1781980503356430 us; error 51 us; skew 500 ppm
I20260620 18:35:03.356922 23091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:03 dist-test-slave-cbn3 krb5kdc[22647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980503, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:03.359949 23091 init.cc:377] Logged in from keytab as kudu/127.10.131.62@KRBTEST.COM (short username kudu)
I20260620 18:35:03.361488 23091 webserver.cc:460] Webserver started at http://127.10.131.62:46269/ using document root <none> and password file <none>
I20260620 18:35:03.362157 23091 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:03.362301 23091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:03.362607 23091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:35:03 dist-test-slave-cbn3 krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980503, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for HTTP/127.10.131.20@KRBTEST.COM
I20260620 18:35:03.428490 23091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "ec32cdc6623e4660b95eaa95035e5b15"
format_stamp: "Formatted at 2026-06-20 18:35:03 on dist-test-slave-cbn3"
server_key: "2f7359ae5d670323becf024000c6a04a"
server_key_iv: "1076b2e4251701cf71589ebb586d3c27"
server_key_version: "kuduclusterkey@0"
I20260620 18:35:03.429418 23091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "ec32cdc6623e4660b95eaa95035e5b15"
format_stamp: "Formatted at 2026-06-20 18:35:03 on dist-test-slave-cbn3"
server_key: "2f7359ae5d670323becf024000c6a04a"
server_key_iv: "1076b2e4251701cf71589ebb586d3c27"
server_key_version: "kuduclusterkey@0"
I20260620 18:35:03.434870 23091 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:35:03.461272 23106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:03.462919 23091 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.008s sys 0.000s
I20260620 18:35:03.463128 23091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "ec32cdc6623e4660b95eaa95035e5b15"
format_stamp: "Formatted at 2026-06-20 18:35:03 on dist-test-slave-cbn3"
server_key: "2f7359ae5d670323becf024000c6a04a"
server_key_iv: "1076b2e4251701cf71589ebb586d3c27"
server_key_version: "kuduclusterkey@0"
I20260620 18:35:03.463354 23091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:03.514997 23091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:03.519142 23091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:03.519471 23091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:03.537701 23091 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:32927
I20260620 18:35:03.537725 23157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:32927 every 8 connection(s)
I20260620 18:35:03.539058 23091 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:35:03.542672 23158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:03.546555 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 23091
I20260620 18:35:03.546763 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:35:03.547451 10764 external_mini_cluster.cc:1428] Setting key 05597384774d290994e5286a2aec8a60
I20260620 18:35:03.554893 23158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15: Bootstrap starting.
I20260620 18:35:03.558552 23158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:03.559948 23158 log.cc:824] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:03.562681 23158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15: No bootstrap required, opened a new log
Jun 20 18:35:03 dist-test-slave-cbn3 krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:35:03.571082 23161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:03.549450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58600 (local address 127.10.131.62:32927)
0620 18:35:03.550254 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:03.550269 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:03.550322 (+ 53us) server_negotiation.cc:407] Connection header received
0620 18:35:03.553186 (+ 2864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:03.553215 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:03.553699 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:03.554129 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:03.555325 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:03.557392 (+ 2067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:03.558662 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:03.559745 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:03.559919 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:03.563936 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:03.563944 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:03.563958 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:03.564026 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:03.567206 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:03.567866 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:03.567883 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:03.567888 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:03.568051 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:03.568417 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:03.568422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:03.568426 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:03.569004 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:03.569316 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:03.569743 (+ 427us) server_negotiation.cc:299] Negotiation successful
0620 18:35:03.570149 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":338,"threads_started":1}
I20260620 18:35:03.570966 23158 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec32cdc6623e4660b95eaa95035e5b15" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 32927 } }
I20260620 18:35:03.571424 23158 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:03.571539 23158 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec32cdc6623e4660b95eaa95035e5b15, State: Initialized, Role: FOLLOWER
I20260620 18:35:03.572109 23158 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec32cdc6623e4660b95eaa95035e5b15" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 32927 } }
I20260620 18:35:03.572369 23158 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:35:03.572551 23158 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:35:03.572760 23158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:03.575465 23158 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec32cdc6623e4660b95eaa95035e5b15" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 32927 } }
I20260620 18:35:03.576031 23158 leader_election.cc:302] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec32cdc6623e4660b95eaa95035e5b15; no voters:
I20260620 18:35:03.576749 23158 leader_election.cc:288] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:35:03.576858 23163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:35:03.577587 23163 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15 [term 1 LEADER]: Becoming Leader. State: Replica: ec32cdc6623e4660b95eaa95035e5b15, State: Running, Role: LEADER
I20260620 18:35:03.578166 23163 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec32cdc6623e4660b95eaa95035e5b15" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 32927 } }
I20260620 18:35:03.578986 23158 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:35:03.580978 23165 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec32cdc6623e4660b95eaa95035e5b15. Latest consensus state: current_term: 1 leader_uuid: "ec32cdc6623e4660b95eaa95035e5b15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec32cdc6623e4660b95eaa95035e5b15" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 32927 } } }
I20260620 18:35:03.581411 23165 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15 [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:03.581652 23164 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec32cdc6623e4660b95eaa95035e5b15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec32cdc6623e4660b95eaa95035e5b15" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 32927 } } }
I20260620 18:35:03.582048 23164 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15 [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:03.583678 23171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:35:03.589294 23171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:35:03.592027 23158 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 18:35:03.606312 23171 catalog_manager.cc:1347] Generated new cluster ID: 20f0e09d3b5c4d9c976d1f1b8e70cb9c
I20260620 18:35:03.606446 23171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:35:03.624436 23171 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:35:03.626379 23171 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:35:03.647856 23171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15: Generated new TSK 0
I20260620 18:35:03.648999 23171 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.131.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab.
Entry for principal kudu/127.10.131.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab.
I20260620 18:35:04.358363 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.131.20:60667/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.1:0
--local_ip_for_outbound_sockets=127.10.131.1
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:32927
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:44799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:35:04 dist-test-slave-cbn3 krb5kdc[22647](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781980504, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 18:35:04.797690 23203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:04.798187 23203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:04.798408 23203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:04.810503 23203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:04.810708 23203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:04.810935 23203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.1
I20260620 18:35:04.824002 23203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:44799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.10.131.20:60667/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:32927
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.1
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:35:04.825773 23203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:04.827157 23203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:04.842479 23214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:04.848119 23216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:04.849159 23213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:04.849614 23203 server_base.cc:1034] running on GCE node
I20260620 18:35:04.850505 23203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:04.853240 23203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:04.854552 23203 hybrid_clock.cc:648] HybridClock initialized: now 1781980504854435 us; error 103 us; skew 500 ppm
I20260620 18:35:04.854916 23203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:04 dist-test-slave-cbn3 krb5kdc[22647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980504, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:04.859035 23203 init.cc:377] Logged in from keytab as kudu/127.10.131.1@KRBTEST.COM (short username kudu)
I20260620 18:35:04.865226 23203 webserver.cc:460] Webserver started at http://127.10.131.1:32921/ using document root <none> and password file <none>
I20260620 18:35:04.866117 23203 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:04.866314 23203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:04.866739 23203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:35:04 dist-test-slave-cbn3 krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980504, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for HTTP/127.10.131.20@KRBTEST.COM
I20260620 18:35:04.887737 23203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-0/data/instance:
uuid: "0e2fe3989ad647519e16decc5fd6611a"
format_stamp: "Formatted at 2026-06-20 18:35:04 on dist-test-slave-cbn3"
server_key: "c4761dc2a1d490bf7c5cbdf0bd18b480"
server_key_iv: "4d4125488abe51134fec57ff5a0b8995"
server_key_version: "kuduclusterkey@0"
I20260620 18:35:04.898622 23203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance:
uuid: "0e2fe3989ad647519e16decc5fd6611a"
format_stamp: "Formatted at 2026-06-20 18:35:04 on dist-test-slave-cbn3"
server_key: "c4761dc2a1d490bf7c5cbdf0bd18b480"
server_key_iv: "4d4125488abe51134fec57ff5a0b8995"
server_key_version: "kuduclusterkey@0"
I20260620 18:35:04.907534 23203 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.000s
I20260620 18:35:04.946416 23223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:04.948295 23203 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.008s sys 0.003s
I20260620 18:35:04.948554 23203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-0/wal
uuid: "0e2fe3989ad647519e16decc5fd6611a"
format_stamp: "Formatted at 2026-06-20 18:35:04 on dist-test-slave-cbn3"
server_key: "c4761dc2a1d490bf7c5cbdf0bd18b480"
server_key_iv: "4d4125488abe51134fec57ff5a0b8995"
server_key_version: "kuduclusterkey@0"
I20260620 18:35:04.949031 23203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:04.981452 23203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:04.986696 23203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:04.987076 23203 kserver.cc:163] Server-wide thread pool size limit: 3276
Jun 20 18:35:04 dist-test-slave-cbn3 krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980504, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for HTTP/127.10.131.20@KRBTEST.COM
I20260620 18:35:05.016481 23203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:05.026969 23203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:05.027141 23203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 18:35:05.027280 23203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:05.027355 23203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:05.097546 23203 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.1:37649
I20260620 18:35:05.097826 23335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.1:37649 every 8 connection(s)
I20260620 18:35:05.099210 23203 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
I20260620 18:35:05.108290 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 23203
I20260620 18:35:05.108528 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance
I20260620 18:35:05.109561 10764 external_mini_cluster.cc:1428] Setting key ee5c37e88bfeba95567697da97329eaa
Jun 20 18:35:05 dist-test-slave-cbn3 krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980504, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:35:05.137632 23338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:05.101968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:36959 (local address 127.10.131.62:32927)
0620 18:35:05.118474 (+ 16506us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:05.118480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:05.118503 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:35:05.118565 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:05.118570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:05.118681 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:05.118803 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:05.120723 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:05.121996 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:05.124278 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:05.124956 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:05.125074 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:05.129546 (+ 4472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:05.129553 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:05.129558 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:05.129599 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:05.133061 (+ 3462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:05.133992 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:05.133997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:05.134001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:05.134147 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:05.134678 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:05.134683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:05.134686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:05.135061 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:05.135198 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:05.137014 (+ 1816us) server_negotiation.cc:299] Negotiation successful
0620 18:35:05.137245 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16373,"thread_start_us":16254,"threads_started":1}
I20260620 18:35:05.139449 23339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:05.102532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:32927 (local address 127.10.131.1:36959)
0620 18:35:05.113904 (+ 11372us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:05.113960 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:05.116285 (+ 2325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:05.119107 (+ 2822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:05.119120 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:05.119737 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:05.120521 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:05.120545 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:05.122155 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:05.122160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:05.124093 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:05.124119 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:05.125354 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:05.125361 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:05.125632 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:05.126419 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:05.126448 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:05.129261 (+ 2813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:05.133226 (+ 3965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:05.133235 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:05.133255 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:05.133833 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:05.134270 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:05.134274 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:05.134278 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:05.134541 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:05.135278 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:05.135286 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:05.135717 (+ 431us) client_negotiation.cc:769] Sending connection context
0620 18:35:05.136046 (+ 329us) client_negotiation.cc:240] Negotiation successful
0620 18:35:05.136583 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11042,"thread_start_us":14061,"threads_started":1}
I20260620 18:35:05.141754 23336 heartbeater.cc:347] Connected to a master server at 127.10.131.62:32927
I20260620 18:35:05.142110 23336 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:05.143077 23336 heartbeater.cc:511] Master 127.10.131.62:32927 requested a full tablet report, sending...
I20260620 18:35:05.145859 23123 ts_manager.cc:194] Registered new tserver with Master: 0e2fe3989ad647519e16decc5fd6611a (127.10.131.1:37649)
I20260620 18:35:05.148209 23123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.1@KRBTEST.COM'} at 127.10.131.1:36959
WARNING: no policy specified for kudu/127.10.131.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab.
Entry for principal kudu/127.10.131.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab.
I20260620 18:35:05.186726 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.131.20:60667/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.2:0
--local_ip_for_outbound_sockets=127.10.131.2
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:32927
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:44799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:35:05 dist-test-slave-cbn3 krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980504, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for HTTP/127.10.131.20@KRBTEST.COM
W20260620 18:35:05.634786 23342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:05.635047 23342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:05.635138 23342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:05.642382 23342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:05.642483 23342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:05.642598 23342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.2
I20260620 18:35:05.652345 23342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:44799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.10.131.20:60667/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:32927
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.2
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:35:05.653573 23342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:05.654989 23342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:05.667738 23357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:05.667809 23359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:05.668220 23356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:05.668568 23342 server_base.cc:1034] running on GCE node
I20260620 18:35:05.669276 23342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:05.670143 23342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:05.671339 23342 hybrid_clock.cc:648] HybridClock initialized: now 1781980505671295 us; error 45 us; skew 500 ppm
I20260620 18:35:05.671643 23342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:05 dist-test-slave-cbn3 krb5kdc[22647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980505, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:05.674790 23342 init.cc:377] Logged in from keytab as kudu/127.10.131.2@KRBTEST.COM (short username kudu)
I20260620 18:35:05.676179 23342 webserver.cc:460] Webserver started at http://127.10.131.2:39237/ using document root <none> and password file <none>
I20260620 18:35:05.676829 23342 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:05.677031 23342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:05.677320 23342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:35:05 dist-test-slave-cbn3 krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980505, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for HTTP/127.10.131.20@KRBTEST.COM
I20260620 18:35:05.692679 23342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-1/data/instance:
uuid: "cf7dca50d06b42a1a2f9ffbbf36e0d72"
format_stamp: "Formatted at 2026-06-20 18:35:05 on dist-test-slave-cbn3"
server_key: "366e32a05dd649cb78a543d5cd3829c4"
server_key_iv: "6543e476aa4fe57c4e342f4cbdc5aa1d"
server_key_version: "kuduclusterkey@0"
I20260620 18:35:05.693521 23342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance:
uuid: "cf7dca50d06b42a1a2f9ffbbf36e0d72"
format_stamp: "Formatted at 2026-06-20 18:35:05 on dist-test-slave-cbn3"
server_key: "366e32a05dd649cb78a543d5cd3829c4"
server_key_iv: "6543e476aa4fe57c4e342f4cbdc5aa1d"
server_key_version: "kuduclusterkey@0"
I20260620 18:35:05.698716 23342 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260620 18:35:05.713291 23366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:05.714848 23342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 18:35:05.715058 23342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-1/wal
uuid: "cf7dca50d06b42a1a2f9ffbbf36e0d72"
format_stamp: "Formatted at 2026-06-20 18:35:05 on dist-test-slave-cbn3"
server_key: "366e32a05dd649cb78a543d5cd3829c4"
server_key_iv: "6543e476aa4fe57c4e342f4cbdc5aa1d"
server_key_version: "kuduclusterkey@0"
I20260620 18:35:05.715294 23342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:05.729674 23342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:05.733259 23342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:05.733549 23342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:05.734475 23342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:05.735786 23342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:05.735857 23342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:05.736029 23342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:05.736119 23342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:05.767076 23342 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.2:39523
I20260620 18:35:05.767109 23478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.2:39523 every 8 connection(s)
I20260620 18:35:05.768370 23342 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
I20260620 18:35:05.777127 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 23342
I20260620 18:35:05.777302 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance
I20260620 18:35:05.777722 10764 external_mini_cluster.cc:1428] Setting key 1c44188a77fc63e1528f69ffe71203ee
Jun 20 18:35:05 dist-test-slave-cbn3 krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980505, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:35:05.788107 23481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:05.770915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:60495 (local address 127.10.131.62:32927)
0620 18:35:05.771368 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:05.771372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:05.772130 (+ 758us) server_negotiation.cc:407] Connection header received
0620 18:35:05.773855 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:05.773861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:05.773964 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:05.774083 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:05.775459 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:05.776224 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:05.777969 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:05.778370 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:05.778526 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:05.782282 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:05.782290 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:05.782300 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:05.782329 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:05.784949 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:05.785667 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:05.785672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:05.785676 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:05.785805 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:05.786236 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:05.786244 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:05.786248 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:05.786665 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:05.786843 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:05.787665 (+ 822us) server_negotiation.cc:299] Negotiation successful
0620 18:35:05.787842 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":243,"threads_started":1}
I20260620 18:35:05.789543 23482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:05.771299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:32927 (local address 127.10.131.2:60495)
0620 18:35:05.771964 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:05.772001 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:05.773586 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:05.774325 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:05.774333 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:05.774735 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:05.775289 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:05.775306 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:05.776365 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:05.776373 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:05.777811 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:05.777843 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:05.778614 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:05.778619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:05.778943 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:05.779745 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:05.779766 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:05.782078 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:05.785084 (+ 3006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:05.785093 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:05.785110 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:05.785537 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:05.785911 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:05.785918 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:05.785921 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:05.786099 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:05.786905 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:05.786911 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:05.787223 (+ 312us) client_negotiation.cc:769] Sending connection context
0620 18:35:05.787544 (+ 321us) client_negotiation.cc:240] Negotiation successful
0620 18:35:05.788113 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":275,"threads_started":1}
I20260620 18:35:05.791083 23479 heartbeater.cc:347] Connected to a master server at 127.10.131.62:32927
I20260620 18:35:05.791384 23479 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:05.792085 23479 heartbeater.cc:511] Master 127.10.131.62:32927 requested a full tablet report, sending...
I20260620 18:35:05.793563 23123 ts_manager.cc:194] Registered new tserver with Master: cf7dca50d06b42a1a2f9ffbbf36e0d72 (127.10.131.2:39523)
I20260620 18:35:05.794351 23123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.2@KRBTEST.COM'} at 127.10.131.2:60495
WARNING: no policy specified for kudu/127.10.131.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab.
Entry for principal kudu/127.10.131.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab.
I20260620 18:35:05.827661 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.131.20:60667/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.3:0
--local_ip_for_outbound_sockets=127.10.131.3
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:32927
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:44799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:06.144657 23485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:06.144933 23485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:06.145097 23485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:06.152398 23485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:06.152539 23485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20260620 18:35:06.152524 23336 heartbeater.cc:503] Master 127.10.131.62:32927 was elected leader, sending a full tablet report...
W20260620 18:35:06.152726 23485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.3
I20260620 18:35:06.162604 23485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:44799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.10.131.20:60667/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:32927
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.3
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:35:06.163817 23485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:06.165174 23485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:06.178450 23490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:06.178416 23493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:06.178493 23491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:06.179149 23485 server_base.cc:1034] running on GCE node
I20260620 18:35:06.179709 23485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:06.180575 23485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:06.181800 23485 hybrid_clock.cc:648] HybridClock initialized: now 1781980506181728 us; error 68 us; skew 500 ppm
I20260620 18:35:06.182134 23485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:06 dist-test-slave-cbn3 krb5kdc[22647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980506, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:06.185197 23485 init.cc:377] Logged in from keytab as kudu/127.10.131.3@KRBTEST.COM (short username kudu)
I20260620 18:35:06.186776 23485 webserver.cc:460] Webserver started at http://127.10.131.3:34259/ using document root <none> and password file <none>
I20260620 18:35:06.187451 23485 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:06.187547 23485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:06.187778 23485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:35:06 dist-test-slave-cbn3 krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980506, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for HTTP/127.10.131.20@KRBTEST.COM
I20260620 18:35:06.199951 23485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-2/data/instance:
uuid: "c67b81f5d2394a0690ae7a9a17d015b3"
format_stamp: "Formatted at 2026-06-20 18:35:06 on dist-test-slave-cbn3"
server_key: "b2faa1947aed5576606143ffcc2aecac"
server_key_iv: "f7b83f52007b10b2ef21977dcc59c534"
server_key_version: "kuduclusterkey@0"
I20260620 18:35:06.200765 23485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance:
uuid: "c67b81f5d2394a0690ae7a9a17d015b3"
format_stamp: "Formatted at 2026-06-20 18:35:06 on dist-test-slave-cbn3"
server_key: "b2faa1947aed5576606143ffcc2aecac"
server_key_iv: "f7b83f52007b10b2ef21977dcc59c534"
server_key_version: "kuduclusterkey@0"
I20260620 18:35:06.206199 23485 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:35:06.223201 23500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:06.224623 23485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:35:06.224800 23485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-2/wal
uuid: "c67b81f5d2394a0690ae7a9a17d015b3"
format_stamp: "Formatted at 2026-06-20 18:35:06 on dist-test-slave-cbn3"
server_key: "b2faa1947aed5576606143ffcc2aecac"
server_key_iv: "f7b83f52007b10b2ef21977dcc59c534"
server_key_version: "kuduclusterkey@0"
I20260620 18:35:06.225099 23485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:06.261633 23485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:06.265314 23485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:06.265609 23485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:06.266700 23485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:06.268172 23485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:06.268250 23485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:06.268391 23485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:06.268440 23485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:06.300611 23485 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.3:42561
I20260620 18:35:06.300653 23612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.3:42561 every 8 connection(s)
I20260620 18:35:06.301868 23485 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
I20260620 18:35:06.309077 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 23485
I20260620 18:35:06.309235 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance
I20260620 18:35:06.309762 10764 external_mini_cluster.cc:1428] Setting key 98d08bbe50c77f5c4a4b69d5e600c686
Jun 20 18:35:06 dist-test-slave-cbn3 krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980506, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:35:06.321507 23615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:06.304297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:54261 (local address 127.10.131.62:32927)
0620 18:35:06.304766 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:06.304771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:06.305736 (+ 965us) server_negotiation.cc:407] Connection header received
0620 18:35:06.307460 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:06.307464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:06.307542 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:06.307635 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:06.308941 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:06.310023 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:06.312079 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:06.312499 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:06.312628 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:06.315909 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:06.315913 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:06.315916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:06.315946 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:06.318574 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:06.319271 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:06.319278 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:06.319286 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:06.319417 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:06.319816 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:06.319825 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:06.319827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:06.320067 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:06.320221 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:06.320915 (+ 694us) server_negotiation.cc:299] Negotiation successful
0620 18:35:06.321133 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":194,"threads_started":1}
I20260620 18:35:06.322335 23616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:06.304954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:32927 (local address 127.10.131.3:54261)
0620 18:35:06.305591 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:06.305626 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:06.307210 (+ 1584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:06.307848 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:06.307855 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:06.308241 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:06.308768 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:06.308783 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:06.310159 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:06.310163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:06.311961 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:06.311974 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:06.312717 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:06.312724 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:06.313000 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:06.313648 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:06.313682 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:06.315700 (+ 2018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:06.318710 (+ 3010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:06.318718 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:06.318733 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:06.319146 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:06.319524 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:06.319530 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:06.319534 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:06.319696 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:06.320191 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:06.320198 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:06.320497 (+ 299us) client_negotiation.cc:769] Sending connection context
0620 18:35:06.320795 (+ 298us) client_negotiation.cc:240] Negotiation successful
0620 18:35:06.321284 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":253,"threads_started":1}
I20260620 18:35:06.323652 23613 heartbeater.cc:347] Connected to a master server at 127.10.131.62:32927
I20260620 18:35:06.323952 23613 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:06.324744 23613 heartbeater.cc:511] Master 127.10.131.62:32927 requested a full tablet report, sending...
I20260620 18:35:06.326042 23123 ts_manager.cc:194] Registered new tserver with Master: c67b81f5d2394a0690ae7a9a17d015b3 (127.10.131.3:42561)
I20260620 18:35:06.326845 23123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.3@KRBTEST.COM'} at 127.10.131.3:54261
I20260620 18:35:06.336974 10764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:35:06.338801 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 23203
Jun 20 18:35:06 dist-test-slave-cbn3 krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.1@KRBTEST.COM
I20260620 18:35:06.355345 23618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:06.339797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55532 (local address 127.10.131.1:37649)
0620 18:35:06.340423 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:06.340430 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:06.340455 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:35:06.340508 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:06.340514 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:06.340702 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:06.340827 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:06.342359 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:06.343343 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:06.344916 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:06.345480 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:06.345589 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:06.349700 (+ 4111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:06.349710 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:06.349722 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:06.349754 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:06.352393 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:06.353240 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:06.353252 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:06.353254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:06.353390 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:06.353763 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:06.353766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:06.353768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:06.354071 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:06.354375 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:06.354733 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 18:35:06.354991 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":319,"threads_started":1}
I20260620 18:35:06.356575 23331 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:06.437338 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 23203
I20260620 18:35:06.453492 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 23342
Jun 20 18:35:06 dist-test-slave-cbn3 krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.2@KRBTEST.COM
I20260620 18:35:06.469481 23620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:06.454379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38762 (local address 127.10.131.2:39523)
0620 18:35:06.454941 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:06.454948 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:06.454973 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:35:06.455032 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:06.455038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:06.455247 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:06.455384 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:06.456624 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:06.457671 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:06.458960 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:06.459497 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:06.459592 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:06.463960 (+ 4368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:06.463971 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:06.463985 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:06.464019 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:06.466647 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:06.467267 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:06.467273 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:06.467275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:06.467399 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:06.467760 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:06.467763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:06.467765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:06.468089 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:06.468387 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:06.468749 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 18:35:06.469056 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":304,"threads_started":1}
I20260620 18:35:06.470666 23474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:06.551008 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 23342
I20260620 18:35:06.566567 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 23485
Jun 20 18:35:06 dist-test-slave-cbn3 krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.3@KRBTEST.COM
I20260620 18:35:06.582511 23622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:06.567579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40086 (local address 127.10.131.3:42561)
0620 18:35:06.568250 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:06.568257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:06.568285 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:35:06.568342 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:06.568348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:06.568572 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:06.568696 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:06.569957 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:06.570942 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:06.572256 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:06.572793 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:06.573153 (+ 360us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:06.577443 (+ 4290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:06.577454 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:06.577468 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:06.577500 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:06.580031 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:06.580714 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:06.580720 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:06.580723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:06.580879 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:06.581207 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:06.581210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:06.581212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:06.581506 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:06.581754 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:06.581945 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 18:35:06.582109 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":371,"threads_started":1}
I20260620 18:35:06.583745 23608 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:06.664813 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 23485
I20260620 18:35:06.680650 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 23091
I20260620 18:35:06.681947 23153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:06.786549 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 23091
I20260620 18:35:06.808182 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:32927
--webserver_interface=127.10.131.62
--webserver_port=46269
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:44799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:32927
--ranger_config_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.131.20:60667/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.131.20:60667/kms with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:07.126780 23625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:07.127199 23625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:07.127375 23625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:07.134662 23625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:35:07.134814 23625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:07.134917 23625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:07.135036 23625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:35:07.135110 23625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:35:07.145213 23625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:44799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.131.20:60667/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:32927
--ranger_config_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:32927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=46269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:35:07.146494 23625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:07.147888 23625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:07.160118 23633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:07.160074 23630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:07.160394 23631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:07.160807 23625 server_base.cc:1034] running on GCE node
I20260620 18:35:07.161617 23625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:07.162968 23625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:07.164234 23625 hybrid_clock.cc:648] HybridClock initialized: now 1781980507164175 us; error 58 us; skew 500 ppm
I20260620 18:35:07.164630 23625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:07 dist-test-slave-cbn3 krb5kdc[22647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980507, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:07.172492 23625 init.cc:377] Logged in from keytab as kudu/127.10.131.62@KRBTEST.COM (short username kudu)
I20260620 18:35:07.174006 23625 webserver.cc:460] Webserver started at http://127.10.131.62:46269/ using document root <none> and password file <none>
I20260620 18:35:07.174674 23625 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:07.174824 23625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 18:35:07 dist-test-slave-cbn3 krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980507, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for HTTP/127.10.131.20@KRBTEST.COM
I20260620 18:35:07.224350 23625 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 18:35:07.227207 23641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:07.228652 23625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 18:35:07.228833 23625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "ec32cdc6623e4660b95eaa95035e5b15"
format_stamp: "Formatted at 2026-06-20 18:35:03 on dist-test-slave-cbn3"
server_key: "2f7359ae5d670323becf024000c6a04a"
server_key_iv: "1076b2e4251701cf71589ebb586d3c27"
server_key_version: "kuduclusterkey@0"
I20260620 18:35:07.229876 23625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:07.257664 23625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:07.261435 23625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:07.261724 23625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:07.279850 23625 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:32927
I20260620 18:35:07.279881 23692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:32927 every 8 connection(s)
I20260620 18:35:07.281342 23625 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:35:07.288057 23693 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:35:07.289948 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 23625
I20260620 18:35:07.290769 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.131.20:60667/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.1:37649
--local_ip_for_outbound_sockets=127.10.131.1
--tserver_master_addrs=127.10.131.62:32927
--webserver_port=32921
--webserver_interface=127.10.131.1
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:44799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.131.20:60667/kms with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:35:07.291527 23693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15: Bootstrap starting.
I20260620 18:35:07.310461 23693 log.cc:824] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:07.319124 23693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:35:07.319657 23693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15: Bootstrap complete.
I20260620 18:35:07.328336 23693 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec32cdc6623e4660b95eaa95035e5b15" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 32927 } }
I20260620 18:35:07.329316 23693 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec32cdc6623e4660b95eaa95035e5b15, State: Initialized, Role: FOLLOWER
I20260620 18:35:07.329900 23693 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec32cdc6623e4660b95eaa95035e5b15" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 32927 } }
I20260620 18:35:07.330142 23693 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:35:07.330271 23693 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:35:07.330466 23693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15 [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:35:07.334463 23693 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec32cdc6623e4660b95eaa95035e5b15" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 32927 } }
I20260620 18:35:07.334949 23693 leader_election.cc:302] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec32cdc6623e4660b95eaa95035e5b15; no voters:
I20260620 18:35:07.335515 23693 leader_election.cc:288] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 18:35:07.335706 23697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 18:35:07.336709 23697 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15 [term 2 LEADER]: Becoming Leader. State: Replica: ec32cdc6623e4660b95eaa95035e5b15, State: Running, Role: LEADER
I20260620 18:35:07.337551 23693 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:35:07.337442 23697 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec32cdc6623e4660b95eaa95035e5b15" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 32927 } }
I20260620 18:35:07.340483 23699 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec32cdc6623e4660b95eaa95035e5b15. Latest consensus state: current_term: 2 leader_uuid: "ec32cdc6623e4660b95eaa95035e5b15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec32cdc6623e4660b95eaa95035e5b15" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 32927 } } }
I20260620 18:35:07.341002 23699 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15 [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:07.341624 23698 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ec32cdc6623e4660b95eaa95035e5b15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec32cdc6623e4660b95eaa95035e5b15" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 32927 } } }
I20260620 18:35:07.341944 23698 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15 [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:07.344261 23704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:35:07.353830 23704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:35:07.355840 23704 catalog_manager.cc:1259] Loaded cluster ID: 20f0e09d3b5c4d9c976d1f1b8e70cb9c
I20260620 18:35:07.356004 23704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:35:07.361088 23704 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:35:07.362969 23704 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15: Loaded TSK: 0
I20260620 18:35:07.364938 23704 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:35:07.370592 23693 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260620 18:35:07.729404 23695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:07.729808 23695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:07.730003 23695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:07.741905 23695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:07.742116 23695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:07.742324 23695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.1
I20260620 18:35:07.758445 23695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:44799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.131.20:60667/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.1:37649
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.131.1
--webserver_port=32921
--tserver_master_addrs=127.10.131.62:32927
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.1
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:35:07.760174 23695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:07.761940 23695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:07.807963 23732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:07.809280 23731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:07.822332 23734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:07.823251 23695 server_base.cc:1034] running on GCE node
I20260620 18:35:07.824167 23695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:07.829138 23695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:07.832686 23695 hybrid_clock.cc:648] HybridClock initialized: now 1781980507832605 us; error 64 us; skew 500 ppm
I20260620 18:35:07.833118 23695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:07 dist-test-slave-cbn3 krb5kdc[22647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980507, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:07.841560 23695 init.cc:377] Logged in from keytab as kudu/127.10.131.1@KRBTEST.COM (short username kudu)
I20260620 18:35:07.858909 23695 webserver.cc:460] Webserver started at http://127.10.131.1:32921/ using document root <none> and password file <none>
I20260620 18:35:07.859709 23695 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:07.859894 23695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 18:35:07 dist-test-slave-cbn3 krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980507, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for HTTP/127.10.131.20@KRBTEST.COM
I20260620 18:35:07.906075 23695 fs_manager.cc:709] Time spent opening directory manager: real 0.020s user 0.005s sys 0.012s
I20260620 18:35:07.925593 23743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:07.929921 23695 fs_manager.cc:725] Time spent opening block manager: real 0.022s user 0.013s sys 0.005s
I20260620 18:35:07.930194 23695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-0/wal
uuid: "0e2fe3989ad647519e16decc5fd6611a"
format_stamp: "Formatted at 2026-06-20 18:35:04 on dist-test-slave-cbn3"
server_key: "c4761dc2a1d490bf7c5cbdf0bd18b480"
server_key_iv: "4d4125488abe51134fec57ff5a0b8995"
server_key_version: "kuduclusterkey@0"
I20260620 18:35:07.933727 23695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:07.986621 23695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:07.992187 23695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:07.992487 23695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:07.998324 23695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:08.000224 23695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:08.000356 23695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:08.000506 23695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:08.000617 23695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:08.041857 23695 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.1:37649
I20260620 18:35:08.041872 23855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.1:37649 every 8 connection(s)
I20260620 18:35:08.043768 23695 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
I20260620 18:35:08.044363 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 23695
I20260620 18:35:08.045208 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.131.20:60667/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.2:39523
--local_ip_for_outbound_sockets=127.10.131.2
--tserver_master_addrs=127.10.131.62:32927
--webserver_port=39237
--webserver_interface=127.10.131.2
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:44799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.131.20:60667/kms with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:35:08 dist-test-slave-cbn3 krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980507, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:35:08.090634 23858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:08.047292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:32927 (local address 127.10.131.1:60795)
0620 18:35:08.058138 (+ 10846us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:08.058187 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:08.066711 (+ 8524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:08.069372 (+ 2661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:08.069383 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:08.070032 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:08.070787 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:08.070814 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:08.073231 (+ 2417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:08.073237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:08.075247 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:08.075268 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:08.077063 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:08.077071 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:08.077338 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:08.078127 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:08.078160 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:08.080958 (+ 2798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:08.085564 (+ 4606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:08.085573 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:08.085590 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:08.086180 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:08.086706 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:08.086711 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:08.086714 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:08.086940 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:08.088008 (+ 1068us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:08.088016 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:08.088409 (+ 393us) client_negotiation.cc:769] Sending connection context
0620 18:35:08.088702 (+ 293us) client_negotiation.cc:240] Negotiation successful
0620 18:35:08.089305 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10543,"thread_start_us":10303,"threads_started":1}
I20260620 18:35:08.092904 23859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:08.048352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:60795 (local address 127.10.131.62:32927)
0620 18:35:08.057473 (+ 9121us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:08.057498 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:08.058706 (+ 1208us) server_negotiation.cc:407] Connection header received
0620 18:35:08.067545 (+ 8839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:08.067568 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:08.068090 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:08.068504 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:08.071035 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:08.073003 (+ 1968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:08.075420 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:08.076429 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:08.076582 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:08.081257 (+ 4675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:08.081268 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:08.081287 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:08.081340 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:08.085388 (+ 4048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:08.086351 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:08.086360 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:08.086368 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:08.086569 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:08.087090 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:08.087096 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:08.087099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:08.087734 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:08.087905 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:08.091307 (+ 3402us) server_negotiation.cc:299] Negotiation successful
0620 18:35:08.091701 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8837,"thread_start_us":8633,"threads_started":1}
I20260620 18:35:08.097373 23856 heartbeater.cc:347] Connected to a master server at 127.10.131.62:32927
I20260620 18:35:08.097697 23856 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:08.098528 23856 heartbeater.cc:511] Master 127.10.131.62:32927 requested a full tablet report, sending...
W20260620 18:35:08.101742 23856 heartbeater.cc:644] Failed to heartbeat to 127.10.131.62:32927 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 18:35:08.101977 23856 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:08.102262 23856 heartbeater.cc:511] Master 127.10.131.62:32927 requested a full tablet report, sending...
I20260620 18:35:08.103497 23856 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:08.103749 23856 heartbeater.cc:511] Master 127.10.131.62:32927 requested a full tablet report, sending...
W20260620 18:35:08.104825 23856 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:35:08.454222 23860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:08.454622 23860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:08.454818 23860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:08.462903 23860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:08.463043 23860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:08.463227 23860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.2
I20260620 18:35:08.478816 23860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:44799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.131.20:60667/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.2:39523
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.131.2
--webserver_port=39237
--tserver_master_addrs=127.10.131.62:32927
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.2
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:35:08.480629 23860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:08.482383 23860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:08.496869 23874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:08.497762 23875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:08.500674 23878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:08.501111 23860 server_base.cc:1034] running on GCE node
I20260620 18:35:08.501832 23860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:08.505803 23860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:08.507119 23860 hybrid_clock.cc:648] HybridClock initialized: now 1781980508507020 us; error 90 us; skew 500 ppm
I20260620 18:35:08.507484 23860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:08 dist-test-slave-cbn3 krb5kdc[22647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980508, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:08.511598 23860 init.cc:377] Logged in from keytab as kudu/127.10.131.2@KRBTEST.COM (short username kudu)
I20260620 18:35:08.558102 23860 webserver.cc:460] Webserver started at http://127.10.131.2:39237/ using document root <none> and password file <none>
I20260620 18:35:08.559227 23860 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:08.559480 23860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 18:35:08 dist-test-slave-cbn3 krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980508, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for HTTP/127.10.131.20@KRBTEST.COM
Jun 20 18:35:08 dist-test-slave-cbn3 krb5kdc[22647](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781980508, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:08.607992 23860 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.008s sys 0.004s
I20260620 18:35:08.616233 23887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:08.618453 23860 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.004s
I20260620 18:35:08.618767 23860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-1/wal
uuid: "cf7dca50d06b42a1a2f9ffbbf36e0d72"
format_stamp: "Formatted at 2026-06-20 18:35:05 on dist-test-slave-cbn3"
server_key: "366e32a05dd649cb78a543d5cd3829c4"
server_key_iv: "6543e476aa4fe57c4e342f4cbdc5aa1d"
server_key_version: "kuduclusterkey@0"
I20260620 18:35:08.620582 23860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:08.671669 23860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:08.677773 23860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:08.678109 23860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:08.689378 23860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:08.698004 23860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:08.698215 23860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:08.698381 23860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:08.698493 23860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:08.926815 23860 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.2:39523
I20260620 18:35:08.926841 24000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.2:39523 every 8 connection(s)
I20260620 18:35:08.928710 23860 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
I20260620 18:35:08.938570 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 23860
I20260620 18:35:08.939540 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.131.20:60667/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.3:42561
--local_ip_for_outbound_sockets=127.10.131.3
--tserver_master_addrs=127.10.131.62:32927
--webserver_port=34259
--webserver_interface=127.10.131.3
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:44799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.131.20:60667/kms with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:35:08 dist-test-slave-cbn3 krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980508, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:35:08.983917 24004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:08.933520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:60387 (local address 127.10.131.62:32927)
0620 18:35:08.957965 (+ 24445us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:08.957973 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:08.962908 (+ 4935us) server_negotiation.cc:407] Connection header received
0620 18:35:08.962994 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:08.963003 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:08.963111 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:08.963225 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:08.965773 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:08.967133 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:08.970067 (+ 2934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:08.970863 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:08.970987 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:08.976447 (+ 5460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:08.976454 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:08.976459 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:08.976504 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:08.980015 (+ 3511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:08.981038 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:08.981045 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:08.981049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:08.981212 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:08.981781 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:08.981790 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:08.981795 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:08.982178 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:08.982319 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:08.983305 (+ 986us) server_negotiation.cc:299] Negotiation successful
0620 18:35:08.983537 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24268,"thread_start_us":29830,"threads_started":1}
I20260620 18:35:08.986477 24003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:08.932712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:32927 (local address 127.10.131.2:60387)
0620 18:35:08.958716 (+ 26004us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:08.958790 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:08.961674 (+ 2884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:08.963800 (+ 2126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:08.963816 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:08.964621 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:08.965533 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:08.965564 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:08.967383 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:08.967390 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:08.969868 (+ 2478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:08.969888 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:08.971407 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:08.971415 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:08.971760 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:08.973133 (+ 1373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:08.973173 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:08.976174 (+ 3001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:08.980211 (+ 4037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:08.980221 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:08.980242 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:08.980896 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:08.981356 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:08.981361 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:08.981366 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:08.981630 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:08.982415 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:08.982424 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:08.982858 (+ 434us) client_negotiation.cc:769] Sending connection context
0620 18:35:08.984477 (+ 1619us) client_negotiation.cc:240] Negotiation successful
0620 18:35:08.985103 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25543,"thread_start_us":29213,"threads_started":1}
I20260620 18:35:08.988456 24001 heartbeater.cc:347] Connected to a master server at 127.10.131.62:32927
I20260620 18:35:08.988780 24001 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:08.989683 24001 heartbeater.cc:511] Master 127.10.131.62:32927 requested a full tablet report, sending...
I20260620 18:35:08.991572 23658 ts_manager.cc:194] Registered new tserver with Master: cf7dca50d06b42a1a2f9ffbbf36e0d72 (127.10.131.2:39523)
I20260620 18:35:08.993571 23658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.2@KRBTEST.COM'} at 127.10.131.2:60387
Jun 20 18:35:09 dist-test-slave-cbn3 krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980508, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for HTTP/127.10.131.20@KRBTEST.COM
I20260620 18:35:09.104997 23856 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:09.105434 23856 heartbeater.cc:511] Master 127.10.131.62:32927 requested a full tablet report, sending...
I20260620 18:35:09.106724 23658 ts_manager.cc:194] Registered new tserver with Master: 0e2fe3989ad647519e16decc5fd6611a (127.10.131.1:37649)
I20260620 18:35:09.107851 23658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.1@KRBTEST.COM'} at 127.10.131.1:60795
Jun 20 18:35:09 dist-test-slave-cbn3 krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980508, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for HTTP/127.10.131.20@KRBTEST.COM
W20260620 18:35:09.413486 24005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:09.414039 24005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:09.414237 24005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:09.425944 24005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:09.426149 24005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:09.426379 24005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.3
I20260620 18:35:09.442484 24005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:44799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.131.20:60667/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.3:42561
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.131.3
--webserver_port=34259
--tserver_master_addrs=127.10.131.62:32927
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.3
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:35:09.444484 24005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:09.446285 24005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:09.478853 24011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:09.479300 24012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:09.484957 24005 server_base.cc:1034] running on GCE node
W20260620 18:35:09.485371 24014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:09.486227 24005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:09.487524 24005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:09.488835 24005 hybrid_clock.cc:648] HybridClock initialized: now 1781980509488756 us; error 65 us; skew 500 ppm
I20260620 18:35:09.489311 24005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:09 dist-test-slave-cbn3 krb5kdc[22647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980509, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:09.493883 24005 init.cc:377] Logged in from keytab as kudu/127.10.131.3@KRBTEST.COM (short username kudu)
I20260620 18:35:09.496780 24005 webserver.cc:460] Webserver started at http://127.10.131.3:34259/ using document root <none> and password file <none>
I20260620 18:35:09.497835 24005 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:09.498020 24005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 18:35:09 dist-test-slave-cbn3 krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980509, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for HTTP/127.10.131.20@KRBTEST.COM
I20260620 18:35:09.534071 24005 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.000s sys 0.004s
I20260620 18:35:09.544258 24022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:09.547394 24005 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.009s sys 0.000s
I20260620 18:35:09.547703 24005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-2/wal
uuid: "c67b81f5d2394a0690ae7a9a17d015b3"
format_stamp: "Formatted at 2026-06-20 18:35:06 on dist-test-slave-cbn3"
server_key: "b2faa1947aed5576606143ffcc2aecac"
server_key_iv: "f7b83f52007b10b2ef21977dcc59c534"
server_key_version: "kuduclusterkey@0"
I20260620 18:35:09.549460 24005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:09.602699 24005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:09.608212 24005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:09.608599 24005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:09.616046 24005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:09.624210 24005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:09.624394 24005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:09.624558 24005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:09.624686 24005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:09.687603 24005 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.3:42561
I20260620 18:35:09.687717 24141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.3:42561 every 8 connection(s)
I20260620 18:35:09.689014 24005 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
I20260620 18:35:09.690564 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 24005
I20260620 18:35:09.713305 24146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:09.692170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57180 (local address 127.10.131.62:32927)
0620 18:35:09.693065 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:09.693074 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:09.693100 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:35:09.693181 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:09.693189 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:09.693306 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:09.696108 (+ 2802us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:09.696367 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:09.698209 (+ 1842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:09.701499 (+ 3290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:09.702730 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:09.703123 (+ 393us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:09.707074 (+ 3951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:09.707078 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:09.707081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:09.707140 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:09.710069 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:09.710968 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:09.710974 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:09.710977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:09.711139 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:09.711629 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:09.711634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:09.711638 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:09.712006 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:09.712277 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:09.712641 (+ 364us) server_negotiation.cc:299] Negotiation successful
0620 18:35:09.712920 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":12665,"threads_started":1}
Jun 20 18:35:09 dist-test-slave-cbn3 krb5kdc[22647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980509, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:35:09.725035 24145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:09.692507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:48257 (local address 127.10.131.62:32927)
0620 18:35:09.704812 (+ 12305us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:09.704818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:09.705751 (+ 933us) server_negotiation.cc:407] Connection header received
0620 18:35:09.708248 (+ 2497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:09.708253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:09.708355 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:09.708463 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:09.710389 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:09.711526 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:09.713927 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:09.714578 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:09.714653 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:09.718727 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:09.718733 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:09.718738 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:09.718788 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:09.722050 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:09.722715 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:09.722721 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:09.722724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:09.722865 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:09.723279 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:09.723284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:09.723287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:09.723619 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:09.723757 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:09.724427 (+ 670us) server_negotiation.cc:299] Negotiation successful
0620 18:35:09.724646 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12186,"thread_start_us":248,"threads_started":1}
I20260620 18:35:09.725801 24147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:09.692444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:32927 (local address 127.10.131.3:48257)
0620 18:35:09.705587 (+ 13143us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:09.705647 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:09.707951 (+ 2304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:09.708742 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:09.708754 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:09.709376 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:09.710167 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:09.710202 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:09.711720 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:09.711724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:09.713712 (+ 1988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:09.713732 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:09.714970 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:09.714979 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:09.715231 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:09.715988 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:09.716018 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:09.718468 (+ 2450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:09.722173 (+ 3705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:09.722180 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:09.722193 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:09.722583 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:09.722965 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:09.722969 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:09.722971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:09.723169 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:09.723730 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:09.723735 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:09.724019 (+ 284us) client_negotiation.cc:769] Sending connection context
0620 18:35:09.724276 (+ 257us) client_negotiation.cc:240] Negotiation successful
0620 18:35:09.724650 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12810,"thread_start_us":12611,"threads_started":1}
I20260620 18:35:09.727264 24142 heartbeater.cc:347] Connected to a master server at 127.10.131.62:32927
I20260620 18:35:09.727614 24142 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:09.728299 24142 heartbeater.cc:511] Master 127.10.131.62:32927 requested a full tablet report, sending...
I20260620 18:35:09.729949 23657 ts_manager.cc:194] Registered new tserver with Master: c67b81f5d2394a0690ae7a9a17d015b3 (127.10.131.3:42561)
I20260620 18:35:09.730768 10764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:35:09.731216 23657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.3@KRBTEST.COM'} at 127.10.131.3:48257
I20260620 18:35:09.748310 24145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:09.734519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57186 (local address 127.10.131.62:32927)
0620 18:35:09.734704 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:09.734710 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:09.734967 (+ 257us) server_negotiation.cc:407] Connection header received
0620 18:35:09.735134 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:09.735139 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:09.735210 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:09.735304 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:09.736740 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:09.737787 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:09.739413 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:09.740084 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:09.740211 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:09.742754 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:09.742761 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:09.742765 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:09.742800 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:09.745926 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:09.746538 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:09.746544 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:09.746548 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:09.746679 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:09.746992 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:09.746997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:09.747001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:09.747319 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:09.747560 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:09.747802 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 18:35:09.748003 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:35:09.754320 23657 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57186:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:35:09.765404 23657 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:35:09.807737 24160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:09.782069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59968 (local address 127.10.131.2:39523)
0620 18:35:09.794320 (+ 12251us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:09.794331 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:09.795120 (+ 789us) server_negotiation.cc:407] Connection header received
0620 18:35:09.795417 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:09.795425 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:09.795660 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:09.795811 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:09.796983 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:09.798371 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:09.805129 (+ 6758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:09.806497 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:09.806626 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:09.806817 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:09.807240 (+ 423us) server_negotiation.cc:299] Negotiation successful
0620 18:35:09.807413 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12058,"thread_start_us":12279,"threads_started":1}
I20260620 18:35:09.807884 24159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:09.781822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:39523 (local address 127.0.0.1:59968)
0620 18:35:09.794991 (+ 13169us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:09.795012 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:09.795235 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:09.796061 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:09.796066 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:09.796089 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:09.796773 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:09.796785 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:09.798565 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:09.798570 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:09.804683 (+ 6113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:09.804697 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:09.806704 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:09.806709 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:09.806954 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:09.807026 (+ 72us) client_negotiation.cc:769] Sending connection context
0620 18:35:09.807193 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 18:35:09.807415 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13004,"mutex_wait_us":2304,"thread_start_us":12846,"threads_started":1}
I20260620 18:35:09.809383 24157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:09.780291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:42561 (local address 127.0.0.1:58872)
0620 18:35:09.780997 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:09.781041 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:09.781198 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:09.794738 (+ 13540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:09.794746 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:09.794786 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:09.795339 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:09.795353 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:09.798262 (+ 2909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:09.798271 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:09.801180 (+ 2909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:09.801195 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:09.808487 (+ 7292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:09.808496 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:09.808699 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:09.808744 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 18:35:09.808823 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 18:35:09.808996 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":504,"threads_started":1}
I20260620 18:35:09.810662 24158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:09.780644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58872 (local address 127.10.131.3:42561)
0620 18:35:09.793518 (+ 12874us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:09.793532 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:09.793583 (+ 51us) server_negotiation.cc:407] Connection header received
0620 18:35:09.793746 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:09.793759 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:09.794141 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:09.795579 (+ 1438us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:09.795926 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:09.798021 (+ 2095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:09.804924 (+ 6903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:09.806490 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:09.810087 (+ 3597us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:09.810254 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:09.810305 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 18:35:09.810408 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12604,"thread_start_us":13719,"threads_started":1}
I20260620 18:35:09.811965 23936 tablet_service.cc:1467] Processing CreateTablet for tablet 4d0147784f984f0b8c1bcdcc10bf68f6 (DEFAULT_TABLE table=test-table [id=3511d284998840b998367612c9322406]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:09.813405 23936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d0147784f984f0b8c1bcdcc10bf68f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:09.813951 24077 tablet_service.cc:1467] Processing CreateTablet for tablet 4d0147784f984f0b8c1bcdcc10bf68f6 (DEFAULT_TABLE table=test-table [id=3511d284998840b998367612c9322406]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:09.815056 24077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d0147784f984f0b8c1bcdcc10bf68f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:09.827973 24161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:09.796016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60192 (local address 127.10.131.1:37649)
0620 18:35:09.799423 (+ 3407us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:09.799431 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:09.800436 (+ 1005us) server_negotiation.cc:407] Connection header received
0620 18:35:09.800797 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:09.800807 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:09.801090 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:09.801504 (+ 414us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:09.802885 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:09.804957 (+ 2072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:09.819209 (+ 14252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:09.820766 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:09.820924 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:09.821125 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:09.827391 (+ 6266us) server_negotiation.cc:299] Negotiation successful
0620 18:35:09.827596 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3259,"thread_start_us":3087,"threads_started":1}
I20260620 18:35:09.828780 24162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:09.799453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:37649 (local address 127.0.0.1:60192)
0620 18:35:09.800339 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:09.800356 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:09.800575 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:09.801836 (+ 1261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:09.801844 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:09.801874 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:09.802620 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:09.802637 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:09.807043 (+ 4406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:09.807048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:09.819006 (+ 11958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:09.819029 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:09.826929 (+ 7900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:09.826936 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:09.827148 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:09.827192 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 18:35:09.828258 (+ 1066us) client_negotiation.cc:240] Negotiation successful
0620 18:35:09.828423 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"thread_start_us":1974,"threads_started":1}
I20260620 18:35:09.832870 23791 tablet_service.cc:1467] Processing CreateTablet for tablet 4d0147784f984f0b8c1bcdcc10bf68f6 (DEFAULT_TABLE table=test-table [id=3511d284998840b998367612c9322406]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:09.834470 23791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d0147784f984f0b8c1bcdcc10bf68f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:09.849570 24163 tablet_bootstrap.cc:492] T 4d0147784f984f0b8c1bcdcc10bf68f6 P cf7dca50d06b42a1a2f9ffbbf36e0d72: Bootstrap starting.
I20260620 18:35:09.851425 24164 tablet_bootstrap.cc:492] T 4d0147784f984f0b8c1bcdcc10bf68f6 P c67b81f5d2394a0690ae7a9a17d015b3: Bootstrap starting.
I20260620 18:35:09.854698 24163 tablet_bootstrap.cc:654] T 4d0147784f984f0b8c1bcdcc10bf68f6 P cf7dca50d06b42a1a2f9ffbbf36e0d72: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:09.855907 24164 tablet_bootstrap.cc:654] T 4d0147784f984f0b8c1bcdcc10bf68f6 P c67b81f5d2394a0690ae7a9a17d015b3: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:09.856267 24163 log.cc:824] T 4d0147784f984f0b8c1bcdcc10bf68f6 P cf7dca50d06b42a1a2f9ffbbf36e0d72: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:09.857543 24164 log.cc:824] T 4d0147784f984f0b8c1bcdcc10bf68f6 P c67b81f5d2394a0690ae7a9a17d015b3: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:09.857834 24165 tablet_bootstrap.cc:492] T 4d0147784f984f0b8c1bcdcc10bf68f6 P 0e2fe3989ad647519e16decc5fd6611a: Bootstrap starting.
I20260620 18:35:09.862540 24165 tablet_bootstrap.cc:654] T 4d0147784f984f0b8c1bcdcc10bf68f6 P 0e2fe3989ad647519e16decc5fd6611a: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:09.863719 24165 log.cc:824] T 4d0147784f984f0b8c1bcdcc10bf68f6 P 0e2fe3989ad647519e16decc5fd6611a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:09.863830 24163 tablet_bootstrap.cc:492] T 4d0147784f984f0b8c1bcdcc10bf68f6 P cf7dca50d06b42a1a2f9ffbbf36e0d72: No bootstrap required, opened a new log
I20260620 18:35:09.864180 24163 ts_tablet_manager.cc:1397] T 4d0147784f984f0b8c1bcdcc10bf68f6 P cf7dca50d06b42a1a2f9ffbbf36e0d72: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.002s
I20260620 18:35:09.871583 24164 tablet_bootstrap.cc:492] T 4d0147784f984f0b8c1bcdcc10bf68f6 P c67b81f5d2394a0690ae7a9a17d015b3: No bootstrap required, opened a new log
I20260620 18:35:09.871865 24164 ts_tablet_manager.cc:1397] T 4d0147784f984f0b8c1bcdcc10bf68f6 P c67b81f5d2394a0690ae7a9a17d015b3: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.007s
I20260620 18:35:09.874586 24165 tablet_bootstrap.cc:492] T 4d0147784f984f0b8c1bcdcc10bf68f6 P 0e2fe3989ad647519e16decc5fd6611a: No bootstrap required, opened a new log
I20260620 18:35:09.874902 24165 ts_tablet_manager.cc:1397] T 4d0147784f984f0b8c1bcdcc10bf68f6 P 0e2fe3989ad647519e16decc5fd6611a: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.005s
I20260620 18:35:09.876892 24163 raft_consensus.cc:348] T 4d0147784f984f0b8c1bcdcc10bf68f6 P cf7dca50d06b42a1a2f9ffbbf36e0d72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67b81f5d2394a0690ae7a9a17d015b3" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42561 } } peers { permanent_uuid: "cf7dca50d06b42a1a2f9ffbbf36e0d72" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39523 } } peers { permanent_uuid: "0e2fe3989ad647519e16decc5fd6611a" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 37649 } }
I20260620 18:35:09.877343 24163 raft_consensus.cc:374] T 4d0147784f984f0b8c1bcdcc10bf68f6 P cf7dca50d06b42a1a2f9ffbbf36e0d72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:09.877445 24163 raft_consensus.cc:729] T 4d0147784f984f0b8c1bcdcc10bf68f6 P cf7dca50d06b42a1a2f9ffbbf36e0d72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf7dca50d06b42a1a2f9ffbbf36e0d72, State: Initialized, Role: FOLLOWER
I20260620 18:35:09.878078 24163 consensus_queue.cc:260] T 4d0147784f984f0b8c1bcdcc10bf68f6 P cf7dca50d06b42a1a2f9ffbbf36e0d72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67b81f5d2394a0690ae7a9a17d015b3" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42561 } } peers { permanent_uuid: "cf7dca50d06b42a1a2f9ffbbf36e0d72" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39523 } } peers { permanent_uuid: "0e2fe3989ad647519e16decc5fd6611a" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 37649 } }
I20260620 18:35:09.883634 24164 raft_consensus.cc:348] T 4d0147784f984f0b8c1bcdcc10bf68f6 P c67b81f5d2394a0690ae7a9a17d015b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67b81f5d2394a0690ae7a9a17d015b3" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42561 } } peers { permanent_uuid: "cf7dca50d06b42a1a2f9ffbbf36e0d72" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39523 } } peers { permanent_uuid: "0e2fe3989ad647519e16decc5fd6611a" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 37649 } }
I20260620 18:35:09.884213 24164 raft_consensus.cc:374] T 4d0147784f984f0b8c1bcdcc10bf68f6 P c67b81f5d2394a0690ae7a9a17d015b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:09.884335 24164 raft_consensus.cc:729] T 4d0147784f984f0b8c1bcdcc10bf68f6 P c67b81f5d2394a0690ae7a9a17d015b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c67b81f5d2394a0690ae7a9a17d015b3, State: Initialized, Role: FOLLOWER
I20260620 18:35:09.885274 24164 consensus_queue.cc:260] T 4d0147784f984f0b8c1bcdcc10bf68f6 P c67b81f5d2394a0690ae7a9a17d015b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67b81f5d2394a0690ae7a9a17d015b3" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42561 } } peers { permanent_uuid: "cf7dca50d06b42a1a2f9ffbbf36e0d72" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39523 } } peers { permanent_uuid: "0e2fe3989ad647519e16decc5fd6611a" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 37649 } }
I20260620 18:35:09.885311 24165 raft_consensus.cc:348] T 4d0147784f984f0b8c1bcdcc10bf68f6 P 0e2fe3989ad647519e16decc5fd6611a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67b81f5d2394a0690ae7a9a17d015b3" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42561 } } peers { permanent_uuid: "cf7dca50d06b42a1a2f9ffbbf36e0d72" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39523 } } peers { permanent_uuid: "0e2fe3989ad647519e16decc5fd6611a" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 37649 } }
I20260620 18:35:09.885799 24165 raft_consensus.cc:374] T 4d0147784f984f0b8c1bcdcc10bf68f6 P 0e2fe3989ad647519e16decc5fd6611a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:09.885913 24165 raft_consensus.cc:729] T 4d0147784f984f0b8c1bcdcc10bf68f6 P 0e2fe3989ad647519e16decc5fd6611a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e2fe3989ad647519e16decc5fd6611a, State: Initialized, Role: FOLLOWER
I20260620 18:35:09.886590 24165 consensus_queue.cc:260] T 4d0147784f984f0b8c1bcdcc10bf68f6 P 0e2fe3989ad647519e16decc5fd6611a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67b81f5d2394a0690ae7a9a17d015b3" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42561 } } peers { permanent_uuid: "cf7dca50d06b42a1a2f9ffbbf36e0d72" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39523 } } peers { permanent_uuid: "0e2fe3989ad647519e16decc5fd6611a" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 37649 } }
I20260620 18:35:09.887810 24001 heartbeater.cc:503] Master 127.10.131.62:32927 was elected leader, sending a full tablet report...
I20260620 18:35:09.888793 24163 ts_tablet_manager.cc:1428] T 4d0147784f984f0b8c1bcdcc10bf68f6 P cf7dca50d06b42a1a2f9ffbbf36e0d72: Time spent starting tablet: real 0.024s user 0.020s sys 0.005s
I20260620 18:35:09.893050 23856 heartbeater.cc:503] Master 127.10.131.62:32927 was elected leader, sending a full tablet report...
I20260620 18:35:09.893718 24165 ts_tablet_manager.cc:1428] T 4d0147784f984f0b8c1bcdcc10bf68f6 P 0e2fe3989ad647519e16decc5fd6611a: Time spent starting tablet: real 0.019s user 0.015s sys 0.000s
I20260620 18:35:09.897567 24142 heartbeater.cc:503] Master 127.10.131.62:32927 was elected leader, sending a full tablet report...
I20260620 18:35:09.898192 24164 ts_tablet_manager.cc:1428] T 4d0147784f984f0b8c1bcdcc10bf68f6 P c67b81f5d2394a0690ae7a9a17d015b3: Time spent starting tablet: real 0.026s user 0.019s sys 0.008s
W20260620 18:35:09.937017 24002 tablet.cc:2369] T 4d0147784f984f0b8c1bcdcc10bf68f6 P cf7dca50d06b42a1a2f9ffbbf36e0d72: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:35:09.940687 24143 tablet.cc:2369] T 4d0147784f984f0b8c1bcdcc10bf68f6 P c67b81f5d2394a0690ae7a9a17d015b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:35:10.069316 23857 tablet.cc:2369] T 4d0147784f984f0b8c1bcdcc10bf68f6 P 0e2fe3989ad647519e16decc5fd6611a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:35:10.145406 24171 raft_consensus.cc:482] T 4d0147784f984f0b8c1bcdcc10bf68f6 P 0e2fe3989ad647519e16decc5fd6611a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:35:10.145730 24171 raft_consensus.cc:504] T 4d0147784f984f0b8c1bcdcc10bf68f6 P 0e2fe3989ad647519e16decc5fd6611a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67b81f5d2394a0690ae7a9a17d015b3" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42561 } } peers { permanent_uuid: "cf7dca50d06b42a1a2f9ffbbf36e0d72" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39523 } } peers { permanent_uuid: "0e2fe3989ad647519e16decc5fd6611a" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 37649 } }
I20260620 18:35:10.147404 24171 leader_election.cc:288] T 4d0147784f984f0b8c1bcdcc10bf68f6 P 0e2fe3989ad647519e16decc5fd6611a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c67b81f5d2394a0690ae7a9a17d015b3 (127.10.131.3:42561), cf7dca50d06b42a1a2f9ffbbf36e0d72 (127.10.131.2:39523)
I20260620 18:35:10.154444 24158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:10.147816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:50171 (local address 127.10.131.3:42561)
0620 18:35:10.148056 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:10.148063 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:10.148475 (+ 412us) server_negotiation.cc:407] Connection header received
0620 18:35:10.148732 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:10.148739 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:10.148820 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:10.148919 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:10.149787 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:10.150610 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:10.152553 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:10.153372 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:10.153577 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:10.153664 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:10.154053 (+ 389us) server_negotiation.cc:299] Negotiation successful
0620 18:35:10.154176 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:35:10.154448 24160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:10.148701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:56879 (local address 127.10.131.2:39523)
0620 18:35:10.148989 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:10.148996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:10.149123 (+ 127us) server_negotiation.cc:407] Connection header received
0620 18:35:10.149245 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:10.149251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:10.149331 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:10.149425 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:10.150054 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:10.150736 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:10.152554 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:10.153367 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:10.153570 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:10.153657 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:10.154029 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 18:35:10.154161 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 18:35:10.154597 24173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:10.148345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:39523 (local address 127.10.131.1:56879)
0620 18:35:10.149022 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:10.149036 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:10.149147 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:10.149485 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:10.149488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:10.149507 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:10.149844 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:10.149877 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:10.150912 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:10.150918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:10.152394 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:10.152407 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:10.153627 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:10.153634 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:10.153844 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:10.153888 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 18:35:10.154004 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 18:35:10.154137 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":61,"thread_start_us":330,"threads_started":1}
I20260620 18:35:10.154636 24172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:10.147699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:42561 (local address 127.10.131.1:50171)
0620 18:35:10.148301 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:10.148321 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:10.148534 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:10.149167 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:10.149172 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:10.149199 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:10.149644 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:10.149656 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:10.150746 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:10.150751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:10.152392 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:10.152404 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:10.153666 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:10.153673 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:10.153901 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:10.153924 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 18:35:10.154028 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 18:35:10.154153 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":144,"thread_start_us":312,"threads_started":1}
I20260620 18:35:10.155597 23956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d0147784f984f0b8c1bcdcc10bf68f6" candidate_uuid: "0e2fe3989ad647519e16decc5fd6611a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf7dca50d06b42a1a2f9ffbbf36e0d72" is_pre_election: true
I20260620 18:35:10.155596 24097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d0147784f984f0b8c1bcdcc10bf68f6" candidate_uuid: "0e2fe3989ad647519e16decc5fd6611a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c67b81f5d2394a0690ae7a9a17d015b3" is_pre_election: true
I20260620 18:35:10.156153 24097 raft_consensus.cc:2454] T 4d0147784f984f0b8c1bcdcc10bf68f6 P c67b81f5d2394a0690ae7a9a17d015b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e2fe3989ad647519e16decc5fd6611a in term 0.
I20260620 18:35:10.156153 23956 raft_consensus.cc:2454] T 4d0147784f984f0b8c1bcdcc10bf68f6 P cf7dca50d06b42a1a2f9ffbbf36e0d72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e2fe3989ad647519e16decc5fd6611a in term 0.
I20260620 18:35:10.156782 23746 leader_election.cc:302] T 4d0147784f984f0b8c1bcdcc10bf68f6 P 0e2fe3989ad647519e16decc5fd6611a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e2fe3989ad647519e16decc5fd6611a, c67b81f5d2394a0690ae7a9a17d015b3; no voters:
I20260620 18:35:10.157231 24171 raft_consensus.cc:2781] T 4d0147784f984f0b8c1bcdcc10bf68f6 P 0e2fe3989ad647519e16decc5fd6611a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:35:10.157385 24171 raft_consensus.cc:482] T 4d0147784f984f0b8c1bcdcc10bf68f6 P 0e2fe3989ad647519e16decc5fd6611a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:35:10.157487 24171 raft_consensus.cc:3037] T 4d0147784f984f0b8c1bcdcc10bf68f6 P 0e2fe3989ad647519e16decc5fd6611a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:10.159751 24171 raft_consensus.cc:504] T 4d0147784f984f0b8c1bcdcc10bf68f6 P 0e2fe3989ad647519e16decc5fd6611a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67b81f5d2394a0690ae7a9a17d015b3" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42561 } } peers { permanent_uuid: "cf7dca50d06b42a1a2f9ffbbf36e0d72" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39523 } } peers { permanent_uuid: "0e2fe3989ad647519e16decc5fd6611a" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 37649 } }
I20260620 18:35:10.160323 24171 leader_election.cc:288] T 4d0147784f984f0b8c1bcdcc10bf68f6 P 0e2fe3989ad647519e16decc5fd6611a [CANDIDATE]: Term 1 election: Requested vote from peers c67b81f5d2394a0690ae7a9a17d015b3 (127.10.131.3:42561), cf7dca50d06b42a1a2f9ffbbf36e0d72 (127.10.131.2:39523)
I20260620 18:35:10.160780 24097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d0147784f984f0b8c1bcdcc10bf68f6" candidate_uuid: "0e2fe3989ad647519e16decc5fd6611a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c67b81f5d2394a0690ae7a9a17d015b3"
I20260620 18:35:10.160919 23956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d0147784f984f0b8c1bcdcc10bf68f6" candidate_uuid: "0e2fe3989ad647519e16decc5fd6611a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf7dca50d06b42a1a2f9ffbbf36e0d72"
I20260620 18:35:10.161057 24097 raft_consensus.cc:3037] T 4d0147784f984f0b8c1bcdcc10bf68f6 P c67b81f5d2394a0690ae7a9a17d015b3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:10.161188 23956 raft_consensus.cc:3037] T 4d0147784f984f0b8c1bcdcc10bf68f6 P cf7dca50d06b42a1a2f9ffbbf36e0d72 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:10.163328 23956 raft_consensus.cc:2454] T 4d0147784f984f0b8c1bcdcc10bf68f6 P cf7dca50d06b42a1a2f9ffbbf36e0d72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e2fe3989ad647519e16decc5fd6611a in term 1.
I20260620 18:35:10.163355 24097 raft_consensus.cc:2454] T 4d0147784f984f0b8c1bcdcc10bf68f6 P c67b81f5d2394a0690ae7a9a17d015b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e2fe3989ad647519e16decc5fd6611a in term 1.
I20260620 18:35:10.163720 23746 leader_election.cc:302] T 4d0147784f984f0b8c1bcdcc10bf68f6 P 0e2fe3989ad647519e16decc5fd6611a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e2fe3989ad647519e16decc5fd6611a, c67b81f5d2394a0690ae7a9a17d015b3; no voters:
I20260620 18:35:10.164007 24171 raft_consensus.cc:2781] T 4d0147784f984f0b8c1bcdcc10bf68f6 P 0e2fe3989ad647519e16decc5fd6611a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:35:10.164422 24171 raft_consensus.cc:686] T 4d0147784f984f0b8c1bcdcc10bf68f6 P 0e2fe3989ad647519e16decc5fd6611a [term 1 LEADER]: Becoming Leader. State: Replica: 0e2fe3989ad647519e16decc5fd6611a, State: Running, Role: LEADER
I20260620 18:35:10.164942 24171 consensus_queue.cc:237] T 4d0147784f984f0b8c1bcdcc10bf68f6 P 0e2fe3989ad647519e16decc5fd6611a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67b81f5d2394a0690ae7a9a17d015b3" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42561 } } peers { permanent_uuid: "cf7dca50d06b42a1a2f9ffbbf36e0d72" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39523 } } peers { permanent_uuid: "0e2fe3989ad647519e16decc5fd6611a" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 37649 } }
I20260620 18:35:10.169298 23657 catalog_manager.cc:5495] T 4d0147784f984f0b8c1bcdcc10bf68f6 P 0e2fe3989ad647519e16decc5fd6611a reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e2fe3989ad647519e16decc5fd6611a (127.10.131.1). New cstate: current_term: 1 leader_uuid: "0e2fe3989ad647519e16decc5fd6611a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67b81f5d2394a0690ae7a9a17d015b3" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf7dca50d06b42a1a2f9ffbbf36e0d72" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e2fe3989ad647519e16decc5fd6611a" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 37649 } health_report { overall_health: HEALTHY } } }
I20260620 18:35:10.185509 24161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:10.180598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60196 (local address 127.10.131.1:37649)
0620 18:35:10.180764 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:10.180768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:10.180805 (+ 37us) server_negotiation.cc:407] Connection header received
0620 18:35:10.180926 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:10.180929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:10.181013 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:10.181124 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:35:10.181443 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:10.182095 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:10.183577 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:10.183986 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:10.184082 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:10.184514 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:35:10.185062 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:35:10.185186 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:10.185244 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 18:35:10.185306 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 18:35:10.193306 24097 raft_consensus.cc:1261] T 4d0147784f984f0b8c1bcdcc10bf68f6 P c67b81f5d2394a0690ae7a9a17d015b3 [term 1 FOLLOWER]: Refusing update from remote peer 0e2fe3989ad647519e16decc5fd6611a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:35:10.193323 23956 raft_consensus.cc:1261] T 4d0147784f984f0b8c1bcdcc10bf68f6 P cf7dca50d06b42a1a2f9ffbbf36e0d72 [term 1 FOLLOWER]: Refusing update from remote peer 0e2fe3989ad647519e16decc5fd6611a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:35:10.194221 24174 consensus_queue.cc:1035] T 4d0147784f984f0b8c1bcdcc10bf68f6 P 0e2fe3989ad647519e16decc5fd6611a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf7dca50d06b42a1a2f9ffbbf36e0d72" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:35:10.194579 24171 consensus_queue.cc:1035] T 4d0147784f984f0b8c1bcdcc10bf68f6 P 0e2fe3989ad647519e16decc5fd6611a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c67b81f5d2394a0690ae7a9a17d015b3" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 42561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:35:10.208035 24179 mvcc.cc:205] Tried to move back new op lower bound from 7298992169742790656 to 7298992169639518208. Current Snapshot: MvccSnapshot[applied={T|T < 7298992169742790656}]
I20260620 18:35:10.225777 23658 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:35:10.226091 23658 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:35:10.230046 23658 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ec32cdc6623e4660b95eaa95035e5b15: Sending DeleteTablet for 3 replicas of tablet 4d0147784f984f0b8c1bcdcc10bf68f6
I20260620 18:35:10.231292 24077 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d0147784f984f0b8c1bcdcc10bf68f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:35:10 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:58872
I20260620 18:35:10.231489 23791 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d0147784f984f0b8c1bcdcc10bf68f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:35:10 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:60192
I20260620 18:35:10.231722 23936 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d0147784f984f0b8c1bcdcc10bf68f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:35:10 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:59968
I20260620 18:35:10.234167 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 23695
I20260620 18:35:10.243970 24188 tablet_replica.cc:330] T 4d0147784f984f0b8c1bcdcc10bf68f6 P 0e2fe3989ad647519e16decc5fd6611a: stopping tablet replica
I20260620 18:35:10.244802 24188 raft_consensus.cc:2229] T 4d0147784f984f0b8c1bcdcc10bf68f6 P 0e2fe3989ad647519e16decc5fd6611a [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:35:10.245786 24188 raft_consensus.cc:2258] T 4d0147784f984f0b8c1bcdcc10bf68f6 P 0e2fe3989ad647519e16decc5fd6611a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:35:10.246685 24187 tablet_replica.cc:330] T 4d0147784f984f0b8c1bcdcc10bf68f6 P c67b81f5d2394a0690ae7a9a17d015b3: stopping tablet replica
I20260620 18:35:10.247207 24187 raft_consensus.cc:2229] T 4d0147784f984f0b8c1bcdcc10bf68f6 P c67b81f5d2394a0690ae7a9a17d015b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:35:10.247716 24187 raft_consensus.cc:2258] T 4d0147784f984f0b8c1bcdcc10bf68f6 P c67b81f5d2394a0690ae7a9a17d015b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:35:10.248028 24188 ts_tablet_manager.cc:1905] T 4d0147784f984f0b8c1bcdcc10bf68f6 P 0e2fe3989ad647519e16decc5fd6611a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:35:10.249905 24187 ts_tablet_manager.cc:1905] T 4d0147784f984f0b8c1bcdcc10bf68f6 P c67b81f5d2394a0690ae7a9a17d015b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:35:10.250726 24189 tablet_replica.cc:330] T 4d0147784f984f0b8c1bcdcc10bf68f6 P cf7dca50d06b42a1a2f9ffbbf36e0d72: stopping tablet replica
I20260620 18:35:10.251178 24189 raft_consensus.cc:2229] T 4d0147784f984f0b8c1bcdcc10bf68f6 P cf7dca50d06b42a1a2f9ffbbf36e0d72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:35:10.251708 24189 raft_consensus.cc:2258] T 4d0147784f984f0b8c1bcdcc10bf68f6 P cf7dca50d06b42a1a2f9ffbbf36e0d72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:35:10.256099 24189 ts_tablet_manager.cc:1905] T 4d0147784f984f0b8c1bcdcc10bf68f6 P cf7dca50d06b42a1a2f9ffbbf36e0d72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:35:10.256141 24188 ts_tablet_manager.cc:1918] T 4d0147784f984f0b8c1bcdcc10bf68f6 P 0e2fe3989ad647519e16decc5fd6611a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:35:10.256429 24188 log.cc:1196] T 4d0147784f984f0b8c1bcdcc10bf68f6 P 0e2fe3989ad647519e16decc5fd6611a: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-0/wal/wals/4d0147784f984f0b8c1bcdcc10bf68f6
I20260620 18:35:10.257126 24188 ts_tablet_manager.cc:1939] T 4d0147784f984f0b8c1bcdcc10bf68f6 P 0e2fe3989ad647519e16decc5fd6611a: Deleting consensus metadata
I20260620 18:35:10.259541 23644 catalog_manager.cc:4841] TS 0e2fe3989ad647519e16decc5fd6611a (127.10.131.1:37649): tablet 4d0147784f984f0b8c1bcdcc10bf68f6 (table test-table [id=3511d284998840b998367612c9322406]) successfully deleted
I20260620 18:35:10.262984 24189 ts_tablet_manager.cc:1918] T 4d0147784f984f0b8c1bcdcc10bf68f6 P cf7dca50d06b42a1a2f9ffbbf36e0d72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:35:10.263092 24187 ts_tablet_manager.cc:1918] T 4d0147784f984f0b8c1bcdcc10bf68f6 P c67b81f5d2394a0690ae7a9a17d015b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:35:10.263295 24189 log.cc:1196] T 4d0147784f984f0b8c1bcdcc10bf68f6 P cf7dca50d06b42a1a2f9ffbbf36e0d72: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-1/wal/wals/4d0147784f984f0b8c1bcdcc10bf68f6
I20260620 18:35:10.263314 24187 log.cc:1196] T 4d0147784f984f0b8c1bcdcc10bf68f6 P c67b81f5d2394a0690ae7a9a17d015b3: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781980314095189-10764-0/minicluster-data/ts-2/wal/wals/4d0147784f984f0b8c1bcdcc10bf68f6
I20260620 18:35:10.263840 24187 ts_tablet_manager.cc:1939] T 4d0147784f984f0b8c1bcdcc10bf68f6 P c67b81f5d2394a0690ae7a9a17d015b3: Deleting consensus metadata
I20260620 18:35:10.263885 24189 ts_tablet_manager.cc:1939] T 4d0147784f984f0b8c1bcdcc10bf68f6 P cf7dca50d06b42a1a2f9ffbbf36e0d72: Deleting consensus metadata
I20260620 18:35:10.265822 23644 catalog_manager.cc:4841] TS cf7dca50d06b42a1a2f9ffbbf36e0d72 (127.10.131.2:39523): tablet 4d0147784f984f0b8c1bcdcc10bf68f6 (table test-table [id=3511d284998840b998367612c9322406]) successfully deleted
I20260620 18:35:10.266147 23644 catalog_manager.cc:4841] TS c67b81f5d2394a0690ae7a9a17d015b3 (127.10.131.3:42561): tablet 4d0147784f984f0b8c1bcdcc10bf68f6 (table test-table [id=3511d284998840b998367612c9322406]) successfully deleted
I20260620 18:35:10.270219 24161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:10.235554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60200 (local address 127.10.131.1:37649)
0620 18:35:10.235780 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:10.235789 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:10.250628 (+ 14839us) server_negotiation.cc:407] Connection header received
0620 18:35:10.252882 (+ 2254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:10.252888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:10.253013 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:10.253100 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:10.255133 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:10.256299 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:10.258294 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:10.258978 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:10.259077 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:10.263006 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:10.263021 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:10.263039 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:10.263087 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:10.266989 (+ 3902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:10.267945 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:10.267954 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:10.267958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:10.268101 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:10.268507 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:10.268510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:10.268512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:10.268941 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:10.269267 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:10.269595 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 18:35:10.269877 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:35:10.270920 23851 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:10.369462 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 23695
I20260620 18:35:10.387511 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 23860
I20260620 18:35:10.401167 24160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:10.388337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59970 (local address 127.10.131.2:39523)
0620 18:35:10.388535 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:10.388541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:10.388589 (+ 48us) server_negotiation.cc:407] Connection header received
0620 18:35:10.388917 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:10.388922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:10.388986 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:10.389097 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:10.390431 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:10.391106 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:10.392376 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:10.392775 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:10.392934 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:10.395430 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:10.395444 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:10.395459 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:10.395494 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:10.398179 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:10.399007 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:10.399017 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:10.399021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:10.399196 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:10.399682 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:10.399688 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:10.399691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:10.400078 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:10.400296 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:10.400572 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 18:35:10.400729 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 18:35:10.401799 23996 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:10.492334 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 23860
I20260620 18:35:10.508726 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 24005
I20260620 18:35:10.521912 24158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:10.509546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58886 (local address 127.10.131.3:42561)
0620 18:35:10.509773 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:10.509778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:10.509798 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:35:10.509861 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:10.509865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:10.509930 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:10.510037 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:10.511338 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:10.512083 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:10.513443 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:10.513833 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:10.513934 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:10.516392 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:10.516406 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:10.516421 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:10.516460 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:10.519386 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:10.520029 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:10.520036 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:10.520038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:10.520158 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:10.520466 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:10.520469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:10.520471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:10.520883 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:10.521109 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:10.521370 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 18:35:10.521523 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260620 18:35:10.522535 24137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:10.610359 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 24005
I20260620 18:35:10.626838 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 23625
I20260620 18:35:10.628050 23688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:10.742251 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 23625
I20260620 18:35:10.765429 10764 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 18:35:11.100570 10764 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 18:35:11.100788 10764 mini_ranger.cc:66] Stopping Ranger...
I20260620 18:35:11.446913 10764 mini_ranger.cc:68] Stopped Ranger
2026-06-20 18:35:11.447 UTC [22677] LOG: received smart shutdown request
2026-06-20 18:35:11.449 UTC [22677] LOG: background worker "logical replication launcher" (PID 22686) exited with exit code 1
2026-06-20 18:35:11.449 UTC [22681] LOG: shutting down
2026-06-20 18:35:11.585 UTC [22677] LOG: database system is shut down
2026-06-20T18:35:11Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51148 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:35:11 dist-test-slave-cbn3 krb5kdc[24203](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:35:11 dist-test-slave-cbn3 krb5kdc[24203](info): set up 2 sockets
Jun 20 18:35:11 dist-test-slave-cbn3 krb5kdc[24203](info): commencing operation
krb5kdc: starting...
W20260620 18:35:12.829241 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:35:12 dist-test-slave-cbn3 krb5kdc[24203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:35:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:35:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.131.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
I20260620 18:35:13.054554 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:45923
--webserver_interface=127.10.131.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:42343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:45923,127.10.131.61:44743,127.10.131.60:36229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:13.373631 24216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:13.373914 24216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:13.374110 24216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:13.381242 24216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:35:13.381336 24216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:13.381438 24216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:13.381572 24216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:35:13.381652 24216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:35:13.391508 24216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:42343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:45923,127.10.131.61:44743,127.10.131.60:36229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:45923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:35:13.392822 24216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:13.394129 24216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:13.406055 24224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:13.406145 24221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:13.406160 24222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:13.407050 24216 server_base.cc:1034] running on GCE node
I20260620 18:35:13.407778 24216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:13.409219 24216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:13.410413 24216 hybrid_clock.cc:648] HybridClock initialized: now 1781980513410353 us; error 47 us; skew 500 ppm
I20260620 18:35:13.410758 24216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:13 dist-test-slave-cbn3 krb5kdc[24203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980513, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:13.416777 24216 init.cc:377] Logged in from keytab as kudu/127.10.131.62@KRBTEST.COM (short username kudu)
I20260620 18:35:13.418303 24216 webserver.cc:460] Webserver started at http://127.10.131.62:36137/ using document root <none> and password file <none>
I20260620 18:35:13.419025 24216 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:13.419171 24216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:13.419489 24216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:13.422629 24216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "96a60e8e0ad542b6af066822eb9c6cd0"
format_stamp: "Formatted at 2026-06-20 18:35:13 on dist-test-slave-cbn3"
server_key: "e18f75ae404d2b7d6cc19e8abb22e3a1"
server_key_iv: "c084306a7613318ee5c0f966f9865c57"
server_key_version: "encryptionkey@0"
I20260620 18:35:13.423450 24216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "96a60e8e0ad542b6af066822eb9c6cd0"
format_stamp: "Formatted at 2026-06-20 18:35:13 on dist-test-slave-cbn3"
server_key: "e18f75ae404d2b7d6cc19e8abb22e3a1"
server_key_iv: "c084306a7613318ee5c0f966f9865c57"
server_key_version: "encryptionkey@0"
I20260620 18:35:13.428622 24216 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260620 18:35:13.432122 24231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:13.433588 24216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:35:13.433777 24216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "96a60e8e0ad542b6af066822eb9c6cd0"
format_stamp: "Formatted at 2026-06-20 18:35:13 on dist-test-slave-cbn3"
server_key: "e18f75ae404d2b7d6cc19e8abb22e3a1"
server_key_iv: "c084306a7613318ee5c0f966f9865c57"
server_key_version: "encryptionkey@0"
I20260620 18:35:13.434003 24216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:13.448203 24216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:13.453279 24216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:13.453521 24216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:13.472775 24216 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:45923
I20260620 18:35:13.472815 24282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:45923 every 8 connection(s)
I20260620 18:35:13.474233 24216 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:35:13.476455 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 24216
I20260620 18:35:13.476684 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:35:13.477455 10764 external_mini_cluster.cc:1428] Setting key cba55f846a67015746ebb4a09108c98b
I20260620 18:35:13.478516 24283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:13.485582 24283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45923 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.131.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.61@KRBTEST.COM" created.
Jun 20 18:35:13 dist-test-slave-cbn3 krb5kdc[24203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980513, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:35:13.508787 24287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:13.488719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52616 (local address 127.10.131.62:45923)
0620 18:35:13.490716 (+ 1997us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:13.490722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:13.490762 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:35:13.493052 (+ 2290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:13.493058 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:13.493304 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:13.493525 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:13.494812 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:13.495969 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:13.497403 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:13.498034 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:13.498236 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:13.501835 (+ 3599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:13.501843 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:13.501855 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:13.501882 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:13.504574 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:13.505310 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:13.505316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:13.505318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:13.505446 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:13.505827 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:13.505837 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:13.505841 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:13.506300 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:13.506503 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:13.507020 (+ 517us) server_negotiation.cc:299] Negotiation successful
0620 18:35:13.507349 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":1796,"spinlock_wait_cycles":17152,"thread_start_us":1534,"threads_started":1}
I20260620 18:35:13.508860 24286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:13.488719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:45923 (local address 127.0.0.1:52616)
0620 18:35:13.489681 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:13.489740 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:13.492607 (+ 2867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:13.493542 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:13.493549 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:13.494287 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:13.494612 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:13.494629 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:13.496126 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:13.496131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:13.497261 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:13.497281 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:13.498275 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:13.498281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:13.498477 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:13.499169 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:13.499203 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:13.501613 (+ 2410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:13.504699 (+ 3086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:13.504710 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:13.504722 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:13.505170 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:13.505561 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:13.505566 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:13.505569 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:13.505714 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:13.506433 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:13.506440 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:13.506650 (+ 210us) client_negotiation.cc:769] Sending connection context
0620 18:35:13.506879 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 18:35:13.507456 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"mutex_wait_us":139,"thread_start_us":386,"threads_started":1}
I20260620 18:35:13.511801 24283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.131.61" port: 44743 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:35:13.513064 24286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:13.512521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:44743 (local address 127.0.0.1:41954)
0620 18:35:13.512737 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:13.512895 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.61:44743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 18:35:13.513375 24232 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.131.61:44743: connect: Connection refused (error 111)
I20260620 18:35:13.514947 24286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:13.514601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:44743 (local address 127.0.0.1:41964)
0620 18:35:13.514748 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:13.514820 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.61:44743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 18:35:13.515246 24283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.61:44743: Network error: Client connection negotiation failed: client connection to 127.10.131.61:44743: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.61.keytab.
Entry for principal kudu/127.10.131.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.61.keytab.
I20260620 18:35:13.526526 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.61:44743
--webserver_interface=127.10.131.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.61.keytab
--principal=kudu/127.10.131.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:42343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:45923,127.10.131.61:44743,127.10.131.60:36229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:35:13.564630 24283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.61" port: 44743 } attempt: 1
I20260620 18:35:13.565892 24286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:13.565454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:44743 (local address 127.0.0.1:41980)
0620 18:35:13.565643 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:13.565760 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.61:44743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 18:35:13.566892 24286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:13.566584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:44743 (local address 127.0.0.1:41992)
0620 18:35:13.566706 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:13.566772 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.61:44743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 18:35:13.567119 24283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.61:44743: Network error: Client connection negotiation failed: client connection to 127.10.131.61:44743: connect: Connection refused (error 111)
I20260620 18:35:13.635360 24283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.61" port: 44743 } attempt: 2
I20260620 18:35:13.636547 24286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:13.636084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:44743 (local address 127.0.0.1:42000)
0620 18:35:13.636325 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:13.636413 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.61:44743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 18:35:13.637655 24286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:13.637330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:44743 (local address 127.0.0.1:42010)
0620 18:35:13.637454 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:13.637529 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.61:44743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 18:35:13.637889 24283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.61:44743: Network error: Client connection negotiation failed: client connection to 127.10.131.61:44743: connect: Connection refused (error 111)
I20260620 18:35:13.729139 24283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.61" port: 44743 } attempt: 3
I20260620 18:35:13.730425 24286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:13.729958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:44743 (local address 127.0.0.1:42026)
0620 18:35:13.730165 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:13.730293 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.61:44743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 18:35:13.731494 24286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:13.731165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:44743 (local address 127.0.0.1:42028)
0620 18:35:13.731293 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:13.731359 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.61:44743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 18:35:13.731751 24283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.61:44743: Network error: Client connection negotiation failed: client connection to 127.10.131.61:44743: connect: Connection refused (error 111)
W20260620 18:35:13.854252 24290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:13.854518 24290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:13.854700 24290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:13.861795 24290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:35:13.861889 24290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:13.861999 24290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:13.862098 24290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:35:13.862191 24290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:35:13.871889 24290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:42343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:45923,127.10.131.61:44743,127.10.131.60:36229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.61.keytab
--principal=kudu/127.10.131.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.61:44743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.131.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:35:13.873204 24290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:13.874445 24290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:35:13.875012 24283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.61" port: 44743 } attempt: 4
I20260620 18:35:13.876226 24286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:13.875695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:44743 (local address 127.0.0.1:42038)
0620 18:35:13.875984 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:13.876078 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.61:44743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260620 18:35:13.877283 24286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:13.876933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:44743 (local address 127.0.0.1:42048)
0620 18:35:13.877083 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:13.877171 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.61:44743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 18:35:13.877471 24283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.61:44743: Network error: Client connection negotiation failed: client connection to 127.10.131.61:44743: connect: Connection refused (error 111)
W20260620 18:35:13.886751 24295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:13.886797 24298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:13.886797 24296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:13.887780 24290 server_base.cc:1034] running on GCE node
I20260620 18:35:13.888535 24290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:13.889967 24290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:13.891196 24290 hybrid_clock.cc:648] HybridClock initialized: now 1781980513891156 us; error 51 us; skew 500 ppm
I20260620 18:35:13.891530 24290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:13 dist-test-slave-cbn3 krb5kdc[24203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980513, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:13.894675 24290 init.cc:377] Logged in from keytab as kudu/127.10.131.61@KRBTEST.COM (short username kudu)
I20260620 18:35:13.896358 24290 webserver.cc:460] Webserver started at http://127.10.131.61:41821/ using document root <none> and password file <none>
I20260620 18:35:13.897132 24290 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:13.897277 24290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:13.897593 24290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:13.900460 24290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-1/data/instance:
uuid: "09436352da2444d982c7c3b83b60250a"
format_stamp: "Formatted at 2026-06-20 18:35:13 on dist-test-slave-cbn3"
server_key: "16335003d448c11a94e7bd4bc720c884"
server_key_iv: "28e0fb732c52f73436bd77d2877c8bbe"
server_key_version: "encryptionkey@0"
I20260620 18:35:13.901278 24290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-1/wal/instance:
uuid: "09436352da2444d982c7c3b83b60250a"
format_stamp: "Formatted at 2026-06-20 18:35:13 on dist-test-slave-cbn3"
server_key: "16335003d448c11a94e7bd4bc720c884"
server_key_iv: "28e0fb732c52f73436bd77d2877c8bbe"
server_key_version: "encryptionkey@0"
I20260620 18:35:13.906347 24290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 18:35:13.909685 24305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:13.910965 24290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:35:13.911172 24290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-1/wal
uuid: "09436352da2444d982c7c3b83b60250a"
format_stamp: "Formatted at 2026-06-20 18:35:13 on dist-test-slave-cbn3"
server_key: "16335003d448c11a94e7bd4bc720c884"
server_key_iv: "28e0fb732c52f73436bd77d2877c8bbe"
server_key_version: "encryptionkey@0"
I20260620 18:35:13.911383 24290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:13.944360 24290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:13.948343 24290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:13.948637 24290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:13.967330 24290 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.61:44743
I20260620 18:35:13.967352 24356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.61:44743 every 8 connection(s)
I20260620 18:35:13.968767 24290 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-1/data/info.pb
I20260620 18:35:13.972558 24357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:13.978785 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 24290
I20260620 18:35:13.978991 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-1/wal/instance
I20260620 18:35:13.979444 10764 external_mini_cluster.cc:1428] Setting key 3c197a29fe62eb30becd9761ed0ae2ae
I20260620 18:35:13.980217 24357 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45923 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:35:13 dist-test-slave-cbn3 krb5kdc[24203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980513, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.61@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.131.60@KRBTEST.COM; defaulting to no policy
I20260620 18:35:14.002818 24287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:13.982819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52626 (local address 127.10.131.62:45923)
0620 18:35:13.983062 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:13.983071 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:13.984285 (+ 1214us) server_negotiation.cc:407] Connection header received
0620 18:35:13.986993 (+ 2708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:13.987002 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:13.987094 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:13.987223 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:13.988622 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:13.989768 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:13.991228 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:13.992010 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:13.992166 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:13.995701 (+ 3535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:13.995711 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:13.995719 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:13.995764 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:13.999650 (+ 3886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:14.000397 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:14.000405 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:14.000412 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:14.000562 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:14.001139 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:14.001144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:14.001148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:14.001482 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:14.001602 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:14.002359 (+ 757us) server_negotiation.cc:299] Negotiation successful
0620 18:35:14.002531 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.60@KRBTEST.COM" created.
I20260620 18:35:14.004014 24359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:13.983283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:45923 (local address 127.0.0.1:52626)
0620 18:35:13.984070 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:13.984122 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:13.986705 (+ 2583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:13.987407 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:13.987414 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:13.987928 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:13.988452 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:13.988467 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:13.989917 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:13.989920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:13.991085 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:13.991096 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:13.992211 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:13.992215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:13.992394 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:13.993139 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:13.993161 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:13.995466 (+ 2305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:13.999806 (+ 4340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:13.999813 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:13.999825 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:14.000246 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:14.000708 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:14.000712 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:14.000714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:14.001004 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:14.001604 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:14.001609 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:14.002000 (+ 391us) client_negotiation.cc:769] Sending connection context
0620 18:35:14.002235 (+ 235us) client_negotiation.cc:240] Negotiation successful
0620 18:35:14.002620 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":313,"threads_started":1}
I20260620 18:35:14.005879 24357 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.131.61" port: 44743 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:35:14 dist-test-slave-cbn3 krb5kdc[24203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980513, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.61@KRBTEST.COM for kudu/127.10.131.61@KRBTEST.COM
I20260620 18:35:14.022320 24359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.006692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:44743 (local address 127.0.0.1:42054)
0620 18:35:14.006918 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:14.006941 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:14.007053 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:14.007990 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:14.007995 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:14.008503 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:14.008724 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:14.008737 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.010106 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.010113 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:14.010952 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:14.010962 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.012172 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.012176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:14.012349 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:14.012988 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:14.013009 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:14.015303 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:14.019778 (+ 4475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:14.019782 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:14.019786 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:14.020189 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:14.020631 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:14.020634 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:14.020637 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:14.020768 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:14.021601 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:14.021605 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:14.021724 (+ 119us) client_negotiation.cc:769] Sending connection context
0620 18:35:14.021837 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 18:35:14.022061 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 18:35:14.022866 24361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.006909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42054 (local address 127.10.131.61:44743)
0620 18:35:14.007574 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:14.007587 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:14.007612 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:35:14.007654 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:14.007659 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:14.007826 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:14.007934 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:14.008943 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.009964 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.011089 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.012025 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.012167 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:14.015525 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:14.015539 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:14.015561 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:14.015612 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:14.019637 (+ 4025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:14.020325 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:14.020336 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:14.020343 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:14.020505 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:14.020908 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:14.020914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:14.020917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:14.021470 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:14.021695 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:14.022020 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 18:35:14.022329 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":385,"threads_started":1}
I20260620 18:35:14.025329 24357 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36229 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:35:14.026634 24359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.026151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:36229 (local address 127.0.0.1:39538)
0620 18:35:14.026304 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:14.026431 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 18:35:14.026962 24307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.131.60:36229: connect: Connection refused (error 111)
I20260620 18:35:14.028553 24359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.028258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:36229 (local address 127.0.0.1:39550)
0620 18:35:14.028393 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:14.028457 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 18:35:14.028760 24357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.60:36229: Network error: Client connection negotiation failed: client connection to 127.10.131.60:36229: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.60.keytab.
Entry for principal kudu/127.10.131.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.60.keytab.
I20260620 18:35:14.032261 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.60:36229
--webserver_interface=127.10.131.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.60.keytab
--principal=kudu/127.10.131.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:42343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:45923,127.10.131.61:44743,127.10.131.60:36229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:35:14.078136 24357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36229 } attempt: 1
I20260620 18:35:14.079448 24359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.078992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:36229 (local address 127.0.0.1:39556)
0620 18:35:14.079191 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:14.079299 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 18:35:14.080621 24359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.080244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:36229 (local address 127.0.0.1:39566)
0620 18:35:14.080426 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:14.080494 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 18:35:14.080900 24357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.60:36229: Network error: Client connection negotiation failed: client connection to 127.10.131.60:36229: connect: Connection refused (error 111)
I20260620 18:35:14.149128 24357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36229 } attempt: 2
I20260620 18:35:14.150264 24359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.149795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:36229 (local address 127.0.0.1:39582)
0620 18:35:14.149997 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:14.150108 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 18:35:14.151304 24359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.150971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:36229 (local address 127.0.0.1:39586)
0620 18:35:14.151134 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:14.151196 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 18:35:14.151583 24357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.60:36229: Network error: Client connection negotiation failed: client connection to 127.10.131.60:36229: connect: Connection refused (error 111)
I20260620 18:35:14.176752 24283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.61" port: 44743 } attempt: 5
Jun 20 18:35:14 dist-test-slave-cbn3 krb5kdc[24203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980513, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for kudu/127.10.131.61@KRBTEST.COM
I20260620 18:35:14.190392 24361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.177828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42060 (local address 127.10.131.61:44743)
0620 18:35:14.178058 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:14.178064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:14.178079 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:35:14.178128 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:14.178132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:14.178209 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:14.178318 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:14.179168 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.179863 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.181126 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.181591 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.181743 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:14.185187 (+ 3444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:14.185192 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:14.185195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:14.185227 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:14.187817 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:14.188441 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:14.188448 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:14.188451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:14.188575 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:14.188977 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:14.188983 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:14.188986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:14.189280 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:14.189436 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:14.189825 (+ 389us) server_negotiation.cc:299] Negotiation successful
0620 18:35:14.190062 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 18:35:14.190392 24286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.177575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:44743 (local address 127.0.0.1:42060)
0620 18:35:14.177764 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:14.177784 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:14.177927 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:14.178367 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:14.178375 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:14.178805 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:14.178995 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:14.179004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.180003 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.180010 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:14.180969 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:14.180980 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.181754 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.181759 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:14.181936 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:14.182663 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:14.182681 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:14.184986 (+ 2305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:14.187952 (+ 2966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:14.187959 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:14.187963 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:14.188320 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:14.188688 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:14.188693 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:14.188697 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:14.188879 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:14.189399 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:14.189404 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:14.189572 (+ 168us) client_negotiation.cc:769] Sending connection context
0620 18:35:14.189711 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 18:35:14.190021 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 18:35:14.191979 24283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36229 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:35:14.193207 24286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.192774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:36229 (local address 127.0.0.1:39594)
0620 18:35:14.192964 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:14.193055 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 18:35:14.194267 24286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.193957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:36229 (local address 127.0.0.1:39604)
0620 18:35:14.194101 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:14.194176 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 18:35:14.194530 24283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.60:36229: Network error: Client connection negotiation failed: client connection to 127.10.131.60:36229: connect: Connection refused (error 111)
I20260620 18:35:14.242813 24357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36229 } attempt: 3
I20260620 18:35:14.244159 24359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.243623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:36229 (local address 127.0.0.1:39610)
0620 18:35:14.243884 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:14.243995 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 18:35:14.245385 24359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.244958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:36229 (local address 127.0.0.1:39622)
0620 18:35:14.245183 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:14.245261 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260620 18:35:14.245594 24357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.60:36229: Network error: Client connection negotiation failed: client connection to 127.10.131.60:36229: connect: Connection refused (error 111)
I20260620 18:35:14.245769 24283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36229 } attempt: 1
I20260620 18:35:14.246788 24286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.246355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:36229 (local address 127.0.0.1:39636)
0620 18:35:14.246564 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:14.246651 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 18:35:14.247852 24286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.247402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:36229 (local address 127.0.0.1:39640)
0620 18:35:14.247588 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:14.247692 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 18:35:14.248103 24283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.60:36229: Network error: Client connection negotiation failed: client connection to 127.10.131.60:36229: connect: Connection refused (error 111)
I20260620 18:35:14.316339 24283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36229 } attempt: 2
I20260620 18:35:14.317693 24286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.317058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:36229 (local address 127.0.0.1:39652)
0620 18:35:14.317345 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:14.317484 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 18:35:14.318710 24286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.318371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:36229 (local address 127.0.0.1:39666)
0620 18:35:14.318536 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:14.318600 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 18:35:14.318948 24283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.60:36229: Network error: Client connection negotiation failed: client connection to 127.10.131.60:36229: connect: Connection refused (error 111)
W20260620 18:35:14.372064 24364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:14.372462 24364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:14.372627 24364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:14.380088 24364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:35:14.380256 24364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:14.380360 24364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:14.380478 24364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:35:14.380568 24364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:35:14.388865 24357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36229 } attempt: 4
I20260620 18:35:14.390110 24359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.389631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:36229 (local address 127.0.0.1:39672)
0620 18:35:14.389872 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:14.389961 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 18:35:14.391158 24359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.390851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:36229 (local address 127.0.0.1:39688)
0620 18:35:14.390988 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:14.391054 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 18:35:14.391402 24357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.60:36229: Network error: Client connection negotiation failed: client connection to 127.10.131.60:36229: connect: Connection refused (error 111)
I20260620 18:35:14.391703 24364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:42343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:45923,127.10.131.61:44743,127.10.131.60:36229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.60.keytab
--principal=kudu/127.10.131.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.60:36229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.131.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:35:14.393146 24364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:14.394459 24364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:14.406899 24372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:14.406975 24370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:14.406950 24369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:14.407805 24364 server_base.cc:1034] running on GCE node
I20260620 18:35:14.408587 24364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:14.409984 24364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:14.411204 24364 hybrid_clock.cc:648] HybridClock initialized: now 1781980514411150 us; error 39 us; skew 500 ppm
I20260620 18:35:14.411556 24364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:14 dist-test-slave-cbn3 krb5kdc[24203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980514, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:14.414897 24364 init.cc:377] Logged in from keytab as kudu/127.10.131.60@KRBTEST.COM (short username kudu)
I20260620 18:35:14.416577 24364 webserver.cc:460] Webserver started at http://127.10.131.60:43367/ using document root <none> and password file <none>
I20260620 18:35:14.417310 24364 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:14.417419 24364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:14.417680 24364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:14.420774 24364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-2/data/instance:
uuid: "3e9d967736254d00b88c77850b39bfb0"
format_stamp: "Formatted at 2026-06-20 18:35:14 on dist-test-slave-cbn3"
server_key: "860f2b1b19129fadd3a7361bfba2f43b"
server_key_iv: "c074a6975f36774d6f58d15f3db0a312"
server_key_version: "encryptionkey@0"
I20260620 18:35:14.421662 24364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-2/wal/instance:
uuid: "3e9d967736254d00b88c77850b39bfb0"
format_stamp: "Formatted at 2026-06-20 18:35:14 on dist-test-slave-cbn3"
server_key: "860f2b1b19129fadd3a7361bfba2f43b"
server_key_iv: "c074a6975f36774d6f58d15f3db0a312"
server_key_version: "encryptionkey@0"
I20260620 18:35:14.425171 24283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36229 } attempt: 3
I20260620 18:35:14.426296 24286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.425803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:36229 (local address 127.0.0.1:39692)
0620 18:35:14.426045 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:14.426133 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 18:35:14.427250 24364 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:35:14.427484 24286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.427007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:36229 (local address 127.0.0.1:39694)
0620 18:35:14.427215 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:14.427310 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260620 18:35:14.427704 24283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.60:36229: Network error: Client connection negotiation failed: client connection to 127.10.131.60:36229: connect: Connection refused (error 111)
I20260620 18:35:14.430785 24379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:14.432448 24364 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 18:35:14.432623 24364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-2/wal
uuid: "3e9d967736254d00b88c77850b39bfb0"
format_stamp: "Formatted at 2026-06-20 18:35:14 on dist-test-slave-cbn3"
server_key: "860f2b1b19129fadd3a7361bfba2f43b"
server_key_iv: "c074a6975f36774d6f58d15f3db0a312"
server_key_version: "encryptionkey@0"
I20260620 18:35:14.432818 24364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:14.444959 24364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:14.448639 24364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:14.449010 24364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:14.466501 24364 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.60:36229
I20260620 18:35:14.466542 24430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.60:36229 every 8 connection(s)
I20260620 18:35:14.467871 24364 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-2/data/info.pb
I20260620 18:35:14.471532 24431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:14.474298 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 24364
I20260620 18:35:14.474493 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781980314095189-10764-0/minicluster-data/master-2/wal/instance
I20260620 18:35:14.474992 10764 external_mini_cluster.cc:1428] Setting key ac2501313338b587f98d1c31d188de11
I20260620 18:35:14.482596 24431 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45923 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:35:14 dist-test-slave-cbn3 krb5kdc[24203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980514, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.60@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
Jun 20 18:35:14 dist-test-slave-cbn3 krb5kdc[24203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.60@KRBTEST.COM
I20260620 18:35:14.504269 24287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.486984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52630 (local address 127.10.131.62:45923)
0620 18:35:14.487155 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:14.487162 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:14.487902 (+ 740us) server_negotiation.cc:407] Connection header received
0620 18:35:14.489513 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:14.489517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:14.489577 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:14.489654 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:14.491008 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.491782 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.493639 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.494292 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.494424 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:14.498083 (+ 3659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:14.498089 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:14.498092 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:14.498123 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:14.501140 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:14.502026 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:14.502029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:14.502032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:14.502157 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:14.502616 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:14.502619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:14.502621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:14.502857 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:14.503039 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:14.503697 (+ 658us) server_negotiation.cc:299] Negotiation successful
0620 18:35:14.503914 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:35:14.505434 24435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.486857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:45923 (local address 127.0.0.1:52630)
0620 18:35:14.487732 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:14.487773 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:14.489274 (+ 1501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:14.489742 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:14.489747 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:14.490477 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:14.490832 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:14.490848 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.491963 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.491968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:14.493505 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:14.493516 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.494515 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.494521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:14.494772 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:14.495564 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:14.495586 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:14.497877 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:14.501274 (+ 3397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:14.501286 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:14.501299 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:14.501906 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:14.502265 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:14.502270 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:14.502273 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:14.502501 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:14.502992 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:14.503000 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:14.503321 (+ 321us) client_negotiation.cc:769] Sending connection context
0620 18:35:14.503573 (+ 252us) client_negotiation.cc:240] Negotiation successful
0620 18:35:14.504083 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"mutex_wait_us":4,"thread_start_us":459,"threads_started":1}
I20260620 18:35:14.505800 24434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.476918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39696 (local address 127.10.131.60:36229)
0620 18:35:14.486697 (+ 9779us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:14.486714 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:14.487128 (+ 414us) server_negotiation.cc:407] Connection header received
0620 18:35:14.489274 (+ 2146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:14.489300 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:14.489592 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:14.489797 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:14.491123 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.492670 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.494343 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.495114 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.495283 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:14.499429 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:14.499437 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:14.499448 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:14.499475 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:14.502705 (+ 3230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:14.503383 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:14.503392 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:14.503395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:14.503526 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:14.503931 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:14.503936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:14.503940 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:14.504288 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:14.504588 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:14.505129 (+ 541us) server_negotiation.cc:299] Negotiation successful
0620 18:35:14.505344 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":9490,"thread_start_us":9229,"threads_started":1}
I20260620 18:35:14.507082 24431 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.131.61" port: 44743 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:35:14 dist-test-slave-cbn3 krb5kdc[24203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980514, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.60@KRBTEST.COM for kudu/127.10.131.61@KRBTEST.COM
Jun 20 18:35:14 dist-test-slave-cbn3 krb5kdc[24203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:35:14.521566 24435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.507824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:44743 (local address 127.0.0.1:42064)
0620 18:35:14.508010 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:14.508030 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:14.508150 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:14.508476 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:14.508479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:14.508956 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:14.509141 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:14.509150 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.510490 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.510494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:14.511734 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:14.511750 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.512615 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.512619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:14.512763 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:14.513385 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:14.513402 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:14.516053 (+ 2651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:14.519273 (+ 3220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:14.519277 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:14.519280 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:14.519638 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:14.520035 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:14.520039 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:14.520041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:14.520178 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:14.520809 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:14.520812 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:14.520994 (+ 182us) client_negotiation.cc:769] Sending connection context
0620 18:35:14.521107 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 18:35:14.521324 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 18:35:14.521874 24361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.507941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42064 (local address 127.10.131.61:44743)
0620 18:35:14.508143 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:14.508149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:14.508167 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:35:14.508260 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:14.508266 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:14.508348 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:14.508453 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:14.509275 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.510351 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.511883 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.512477 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.512603 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:14.516234 (+ 3631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:14.516239 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:14.516242 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:14.516275 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:14.519130 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:14.519737 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:14.519742 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:14.519746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:14.519913 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:14.520286 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:14.520293 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:14.520299 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:14.520685 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:14.520873 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:14.521234 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 18:35:14.521472 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:35:14.522747 24431 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36229 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:35:14.525293 24287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.508982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52636 (local address 127.10.131.62:45923)
0620 18:35:14.509142 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:14.509147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:14.509163 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:35:14.509308 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:14.509312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:14.509364 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:14.509438 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:14.510943 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.511990 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.513671 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.514317 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.514477 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:14.518773 (+ 4296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:14.518781 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:14.518785 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:14.518824 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:14.522194 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:14.523035 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:14.523040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:14.523043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:14.523196 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:14.523662 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:14.523668 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:14.523672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:14.524112 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:14.524390 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:14.524700 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 18:35:14.524960 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 18:35:14 dist-test-slave-cbn3 krb5kdc[24203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980514, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.60@KRBTEST.COM for kudu/127.10.131.60@KRBTEST.COM
Jun 20 18:35:14 dist-test-slave-cbn3 krb5kdc[24203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.61@KRBTEST.COM
I20260620 18:35:14.536226 24435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.523752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:36229 (local address 127.0.0.1:39702)
0620 18:35:14.523909 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:14.523928 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:14.524033 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:14.524370 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:14.524374 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:14.524736 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:14.524979 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:14.524988 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.526018 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.526025 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:14.527266 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:14.527284 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.528026 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.528033 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:14.528246 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:14.528799 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:14.528828 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:14.530778 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:14.533774 (+ 2996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:14.533780 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:14.533784 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:14.534147 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:14.534514 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:14.534519 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:14.534523 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:14.534683 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:14.535174 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:14.535179 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:14.535318 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 18:35:14.535472 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 18:35:14.535794 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":1}
I20260620 18:35:14.536247 24434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.523629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39702 (local address 127.10.131.60:36229)
0620 18:35:14.523821 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:14.523830 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:14.523997 (+ 167us) server_negotiation.cc:407] Connection header received
0620 18:35:14.524156 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:14.524160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:14.524212 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:14.524345 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:14.525102 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.525903 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.527422 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.527865 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.527961 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:14.530960 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:14.530965 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:14.530968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:14.530992 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:14.533632 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:14.534268 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:14.534271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:14.534274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:14.534397 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:14.534812 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:14.534815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:14.534817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:14.535058 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:14.535207 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:14.535549 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 18:35:14.535969 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":204,"server-negotiator.queue_time_us":65}
I20260620 18:35:14.541252 24361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.526875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42080 (local address 127.10.131.61:44743)
0620 18:35:14.527034 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:14.527038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:14.527052 (+ 14us) server_negotiation.cc:407] Connection header received
0620 18:35:14.527210 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:14.527214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:14.527265 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:14.527338 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:14.528937 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.529614 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.530918 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.531340 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.531493 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:14.535639 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:14.535643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:14.535647 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:14.535676 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:14.538673 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:14.539336 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:14.539342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:14.539345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:14.539488 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:14.539855 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:14.539861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:14.539864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:14.540120 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:14.540390 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:14.540675 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 18:35:14.540950 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:35:14.541322 24431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e9d967736254d00b88c77850b39bfb0: Bootstrap starting.
I20260620 18:35:14.545331 24431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e9d967736254d00b88c77850b39bfb0: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:14.546043 10764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:35:14.546588 24431 log.cc:824] T 00000000000000000000000000000000 P 3e9d967736254d00b88c77850b39bfb0: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:14.549181 24431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e9d967736254d00b88c77850b39bfb0: No bootstrap required, opened a new log
I20260620 18:35:14.562800 24431 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e9d967736254d00b88c77850b39bfb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a60e8e0ad542b6af066822eb9c6cd0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45923 } } peers { permanent_uuid: "09436352da2444d982c7c3b83b60250a" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 44743 } } peers { permanent_uuid: "3e9d967736254d00b88c77850b39bfb0" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36229 } }
I20260620 18:35:14.563325 24431 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e9d967736254d00b88c77850b39bfb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:14.563433 24431 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e9d967736254d00b88c77850b39bfb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e9d967736254d00b88c77850b39bfb0, State: Initialized, Role: FOLLOWER
I20260620 18:35:14.564323 24431 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e9d967736254d00b88c77850b39bfb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a60e8e0ad542b6af066822eb9c6cd0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45923 } } peers { permanent_uuid: "09436352da2444d982c7c3b83b60250a" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 44743 } } peers { permanent_uuid: "3e9d967736254d00b88c77850b39bfb0" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36229 } }
I20260620 18:35:14.570933 24444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e9d967736254d00b88c77850b39bfb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a60e8e0ad542b6af066822eb9c6cd0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45923 } } peers { permanent_uuid: "09436352da2444d982c7c3b83b60250a" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 44743 } } peers { permanent_uuid: "3e9d967736254d00b88c77850b39bfb0" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36229 } } }
I20260620 18:35:14.571462 24444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e9d967736254d00b88c77850b39bfb0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:35:14.572414 24431 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e9d967736254d00b88c77850b39bfb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:35:14.572732 24361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.551518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42084 (local address 127.10.131.61:44743)
0620 18:35:14.551693 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:14.551700 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:14.552142 (+ 442us) server_negotiation.cc:407] Connection header received
0620 18:35:14.552459 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:14.552468 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:14.552564 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:14.552647 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:14.555757 (+ 3110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.557207 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.560173 (+ 2966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.561099 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.561226 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:14.565547 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:14.565555 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:14.565562 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:14.565609 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:14.569197 (+ 3588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:14.570105 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:14.570113 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:14.570117 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:14.570286 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:14.570756 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:14.570760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:14.570763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:14.571132 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:14.571361 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:14.572078 (+ 717us) server_negotiation.cc:299] Negotiation successful
0620 18:35:14.572336 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 18:35:14.584359 24434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.551495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39712 (local address 127.10.131.60:36229)
0620 18:35:14.551660 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:14.551666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:14.564021 (+ 12355us) server_negotiation.cc:407] Connection header received
0620 18:35:14.564476 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:14.564484 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:14.564570 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:14.564641 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:14.566605 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.567762 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.571742 (+ 3980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.572454 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.572570 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:14.576854 (+ 4284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:14.576860 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:14.576865 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:14.576903 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:14.580994 (+ 4091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:14.581853 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:14.581859 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:14.581862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:14.582045 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:14.582512 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:14.582517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:14.582520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:14.582922 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:14.583165 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:14.583709 (+ 544us) server_negotiation.cc:299] Negotiation successful
0620 18:35:14.583954 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 18:35:14.588436 24287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.551099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52644 (local address 127.10.131.62:45923)
0620 18:35:14.551337 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:14.551343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:14.570485 (+ 19142us) server_negotiation.cc:407] Connection header received
0620 18:35:14.570748 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:14.570753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:14.570857 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:14.570930 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:14.572850 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.573847 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.576688 (+ 2841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.577394 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.577524 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:14.580643 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:14.580650 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:14.580655 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:14.580700 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:14.584680 (+ 3980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:14.585772 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:14.585780 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:14.585786 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:14.586001 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:14.586512 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:14.586519 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:14.586525 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:14.586948 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:14.587259 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:14.587587 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 18:35:14.587905 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 18:35:14.583079 24364 master_runner.cc:186] Error getting master registration for 127.10.131.62:45923: OK, instance_id { permanent_uuid: "96a60e8e0ad542b6af066822eb9c6cd0" instance_seqno: 1781980513467261 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:35:14.592948 24364 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 18:35:14.595423 24455 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3e9d967736254d00b88c77850b39bfb0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:35:14.595551 24455 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:35:14.604959 24283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36229 } attempt: 4
Jun 20 18:35:14 dist-test-slave-cbn3 krb5kdc[24203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980513, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for kudu/127.10.131.60@KRBTEST.COM
I20260620 18:35:14.620802 24434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.605592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39722 (local address 127.10.131.60:36229)
0620 18:35:14.605750 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:14.605756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:14.605774 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:35:14.605952 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:14.605955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:14.606024 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:14.606100 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:14.607013 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.607890 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.609853 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.610355 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.610477 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:14.614274 (+ 3797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:14.614279 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:14.614282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:14.614322 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:14.618095 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:14.618888 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:14.618893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:14.618896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:14.619084 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:14.619520 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:14.619527 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:14.619529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:14.619845 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:14.619952 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:14.620382 (+ 430us) server_negotiation.cc:299] Negotiation successful
0620 18:35:14.620542 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:35:14.620995 24286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.605497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:36229 (local address 127.0.0.1:39722)
0620 18:35:14.605709 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:14.605730 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:14.605837 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:14.606161 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:14.606167 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:14.606672 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:14.606883 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:14.606896 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.608026 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.608032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:14.609573 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:14.609589 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.610537 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.610543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:14.610742 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:14.611403 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:14.611427 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:14.614100 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:14.618253 (+ 4153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:14.618259 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:14.618263 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:14.618789 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:14.619214 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:14.619219 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:14.619223 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:14.619399 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:14.619962 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:14.619967 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:14.620151 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 18:35:14.620304 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 18:35:14.620591 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 18:35:14.625447 24283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96a60e8e0ad542b6af066822eb9c6cd0: Bootstrap starting.
I20260620 18:35:14.629509 24283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96a60e8e0ad542b6af066822eb9c6cd0: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:14.630525 24283 log.cc:824] T 00000000000000000000000000000000 P 96a60e8e0ad542b6af066822eb9c6cd0: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:14.633109 24283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96a60e8e0ad542b6af066822eb9c6cd0: No bootstrap required, opened a new log
I20260620 18:35:14.640908 24283 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96a60e8e0ad542b6af066822eb9c6cd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a60e8e0ad542b6af066822eb9c6cd0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45923 } } peers { permanent_uuid: "09436352da2444d982c7c3b83b60250a" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 44743 } } peers { permanent_uuid: "3e9d967736254d00b88c77850b39bfb0" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36229 } }
I20260620 18:35:14.641201 24283 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96a60e8e0ad542b6af066822eb9c6cd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:14.641314 24283 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96a60e8e0ad542b6af066822eb9c6cd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96a60e8e0ad542b6af066822eb9c6cd0, State: Initialized, Role: FOLLOWER
I20260620 18:35:14.641844 24283 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96a60e8e0ad542b6af066822eb9c6cd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a60e8e0ad542b6af066822eb9c6cd0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45923 } } peers { permanent_uuid: "09436352da2444d982c7c3b83b60250a" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 44743 } } peers { permanent_uuid: "3e9d967736254d00b88c77850b39bfb0" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36229 } }
I20260620 18:35:14.642812 24458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96a60e8e0ad542b6af066822eb9c6cd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a60e8e0ad542b6af066822eb9c6cd0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45923 } } peers { permanent_uuid: "09436352da2444d982c7c3b83b60250a" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 44743 } } peers { permanent_uuid: "3e9d967736254d00b88c77850b39bfb0" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36229 } } }
I20260620 18:35:14.643210 24458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96a60e8e0ad542b6af066822eb9c6cd0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:35:14.643958 24283 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96a60e8e0ad542b6af066822eb9c6cd0 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 18:35:14.653991 24469 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 96a60e8e0ad542b6af066822eb9c6cd0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:35:14.654191 24469 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:35:14.663076 24444 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e9d967736254d00b88c77850b39bfb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:35:14.663390 24444 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e9d967736254d00b88c77850b39bfb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a60e8e0ad542b6af066822eb9c6cd0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45923 } } peers { permanent_uuid: "09436352da2444d982c7c3b83b60250a" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 44743 } } peers { permanent_uuid: "3e9d967736254d00b88c77850b39bfb0" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36229 } }
I20260620 18:35:14.664073 24458 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96a60e8e0ad542b6af066822eb9c6cd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:35:14.664345 24458 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96a60e8e0ad542b6af066822eb9c6cd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a60e8e0ad542b6af066822eb9c6cd0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45923 } } peers { permanent_uuid: "09436352da2444d982c7c3b83b60250a" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 44743 } } peers { permanent_uuid: "3e9d967736254d00b88c77850b39bfb0" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36229 } }
I20260620 18:35:14.664973 24444 leader_election.cc:288] T 00000000000000000000000000000000 P 3e9d967736254d00b88c77850b39bfb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96a60e8e0ad542b6af066822eb9c6cd0 (127.10.131.62:45923), 09436352da2444d982c7c3b83b60250a (127.10.131.61:44743)
I20260620 18:35:14.665746 24458 leader_election.cc:288] T 00000000000000000000000000000000 P 96a60e8e0ad542b6af066822eb9c6cd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09436352da2444d982c7c3b83b60250a (127.10.131.61:44743), 3e9d967736254d00b88c77850b39bfb0 (127.10.131.60:36229)
I20260620 18:35:14.665601 24258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e9d967736254d00b88c77850b39bfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96a60e8e0ad542b6af066822eb9c6cd0" is_pre_election: true
I20260620 18:35:14.666042 24258 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 96a60e8e0ad542b6af066822eb9c6cd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e9d967736254d00b88c77850b39bfb0 in term 0.
I20260620 18:35:14.666828 24382 leader_election.cc:302] T 00000000000000000000000000000000 P 3e9d967736254d00b88c77850b39bfb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e9d967736254d00b88c77850b39bfb0, 96a60e8e0ad542b6af066822eb9c6cd0; no voters:
I20260620 18:35:14.667443 24406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96a60e8e0ad542b6af066822eb9c6cd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e9d967736254d00b88c77850b39bfb0" is_pre_election: true
I20260620 18:35:14.667879 24406 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3e9d967736254d00b88c77850b39bfb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96a60e8e0ad542b6af066822eb9c6cd0 in term 0.
I20260620 18:35:14.668524 24233 leader_election.cc:302] T 00000000000000000000000000000000 P 96a60e8e0ad542b6af066822eb9c6cd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e9d967736254d00b88c77850b39bfb0, 96a60e8e0ad542b6af066822eb9c6cd0; no voters:
I20260620 18:35:14.668963 24458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96a60e8e0ad542b6af066822eb9c6cd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:35:14.669109 24458 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96a60e8e0ad542b6af066822eb9c6cd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:35:14.669203 24458 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96a60e8e0ad542b6af066822eb9c6cd0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:14.653463 24216 master_runner.cc:186] Error getting master registration for 127.10.131.61:44743: OK, instance_id { permanent_uuid: "09436352da2444d982c7c3b83b60250a" instance_seqno: 1781980513961868 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:35:14.671507 24216 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:35:14.672461 24444 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e9d967736254d00b88c77850b39bfb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:35:14.672624 24444 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e9d967736254d00b88c77850b39bfb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:35:14.672744 24444 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e9d967736254d00b88c77850b39bfb0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:14.673372 24458 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96a60e8e0ad542b6af066822eb9c6cd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a60e8e0ad542b6af066822eb9c6cd0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45923 } } peers { permanent_uuid: "09436352da2444d982c7c3b83b60250a" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 44743 } } peers { permanent_uuid: "3e9d967736254d00b88c77850b39bfb0" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36229 } }
I20260620 18:35:14.676128 24444 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e9d967736254d00b88c77850b39bfb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a60e8e0ad542b6af066822eb9c6cd0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45923 } } peers { permanent_uuid: "09436352da2444d982c7c3b83b60250a" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 44743 } } peers { permanent_uuid: "3e9d967736254d00b88c77850b39bfb0" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36229 } }
I20260620 18:35:14.676570 24406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96a60e8e0ad542b6af066822eb9c6cd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e9d967736254d00b88c77850b39bfb0"
I20260620 18:35:14.676896 24406 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3e9d967736254d00b88c77850b39bfb0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 96a60e8e0ad542b6af066822eb9c6cd0 in current term 1: Already voted for candidate 3e9d967736254d00b88c77850b39bfb0 in this term.
I20260620 18:35:14.677395 24258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e9d967736254d00b88c77850b39bfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96a60e8e0ad542b6af066822eb9c6cd0"
I20260620 18:35:14.677685 24258 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 96a60e8e0ad542b6af066822eb9c6cd0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3e9d967736254d00b88c77850b39bfb0 in current term 1: Already voted for candidate 96a60e8e0ad542b6af066822eb9c6cd0 in this term.
I20260620 18:35:14.677798 24458 leader_election.cc:288] T 00000000000000000000000000000000 P 96a60e8e0ad542b6af066822eb9c6cd0 [CANDIDATE]: Term 1 election: Requested vote from peers 09436352da2444d982c7c3b83b60250a (127.10.131.61:44743), 3e9d967736254d00b88c77850b39bfb0 (127.10.131.60:36229)
I20260620 18:35:14.678550 24444 leader_election.cc:288] T 00000000000000000000000000000000 P 3e9d967736254d00b88c77850b39bfb0 [CANDIDATE]: Term 1 election: Requested vote from peers 96a60e8e0ad542b6af066822eb9c6cd0 (127.10.131.62:45923), 09436352da2444d982c7c3b83b60250a (127.10.131.61:44743)
I20260620 18:35:14.690694 24357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36229 } attempt: 5
Jun 20 18:35:14 dist-test-slave-cbn3 krb5kdc[24203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980513, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.61@KRBTEST.COM for kudu/127.10.131.60@KRBTEST.COM
W20260620 18:35:14.700969 24385 tablet.cc:2369] T 00000000000000000000000000000000 P 3e9d967736254d00b88c77850b39bfb0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:35:14.706398 24237 tablet.cc:2369] T 00000000000000000000000000000000 P 96a60e8e0ad542b6af066822eb9c6cd0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:35:14.708647 24359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.691212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:36229 (local address 127.0.0.1:39736)
0620 18:35:14.691579 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:14.691600 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:14.691708 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:14.692030 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:14.692035 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:14.692546 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:14.692773 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:14.692785 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.694453 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.694460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:14.695954 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:14.695969 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.697194 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.697200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:14.697432 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:14.698150 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:14.698174 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:14.701219 (+ 3045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:14.705560 (+ 4341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:14.705566 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:14.705570 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:14.706112 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:14.706659 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:14.706665 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:14.706669 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:14.706846 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:14.707583 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:14.707589 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:14.707762 (+ 173us) client_negotiation.cc:769] Sending connection context
0620 18:35:14.707919 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 18:35:14.708224 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234}
I20260620 18:35:14.710078 24434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:14.691545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39736 (local address 127.10.131.60:36229)
0620 18:35:14.691712 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:14.691718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:14.691737 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:35:14.691816 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:14.691823 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:14.691897 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:14.691993 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:14.693014 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.694170 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.696128 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:14.696818 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:14.696979 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:14.701472 (+ 4493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:14.701480 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:14.701485 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:14.701531 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:14.705369 (+ 3838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:14.706260 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:14.706267 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:14.706273 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:14.706418 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:14.706979 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:14.706984 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:14.706988 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:14.707373 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:14.707522 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:14.709293 (+ 1771us) server_negotiation.cc:299] Negotiation successful
0620 18:35:14.709686 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 18:35:14.716189 24357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09436352da2444d982c7c3b83b60250a: Bootstrap starting.
I20260620 18:35:14.723086 24357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09436352da2444d982c7c3b83b60250a: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:14.724471 24357 log.cc:824] T 00000000000000000000000000000000 P 09436352da2444d982c7c3b83b60250a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:14.734535 24357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09436352da2444d982c7c3b83b60250a: No bootstrap required, opened a new log
I20260620 18:35:14.746287 24357 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09436352da2444d982c7c3b83b60250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a60e8e0ad542b6af066822eb9c6cd0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45923 } } peers { permanent_uuid: "09436352da2444d982c7c3b83b60250a" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 44743 } } peers { permanent_uuid: "3e9d967736254d00b88c77850b39bfb0" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36229 } }
I20260620 18:35:14.746737 24357 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09436352da2444d982c7c3b83b60250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:14.746901 24357 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09436352da2444d982c7c3b83b60250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09436352da2444d982c7c3b83b60250a, State: Initialized, Role: FOLLOWER
I20260620 18:35:14.747601 24357 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09436352da2444d982c7c3b83b60250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a60e8e0ad542b6af066822eb9c6cd0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45923 } } peers { permanent_uuid: "09436352da2444d982c7c3b83b60250a" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 44743 } } peers { permanent_uuid: "3e9d967736254d00b88c77850b39bfb0" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36229 } }
I20260620 18:35:14.750345 24357 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09436352da2444d982c7c3b83b60250a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:35:14.750319 24331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96a60e8e0ad542b6af066822eb9c6cd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09436352da2444d982c7c3b83b60250a" is_pre_election: true
I20260620 18:35:14.750893 24331 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 09436352da2444d982c7c3b83b60250a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96a60e8e0ad542b6af066822eb9c6cd0 in term 0.
I20260620 18:35:14.751305 24329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e9d967736254d00b88c77850b39bfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09436352da2444d982c7c3b83b60250a"
I20260620 18:35:14.751551 24329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09436352da2444d982c7c3b83b60250a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:14.754989 24329 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 09436352da2444d982c7c3b83b60250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e9d967736254d00b88c77850b39bfb0 in term 1.
I20260620 18:35:14.755147 24330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96a60e8e0ad542b6af066822eb9c6cd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09436352da2444d982c7c3b83b60250a"
I20260620 18:35:14.755366 24332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e9d967736254d00b88c77850b39bfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09436352da2444d982c7c3b83b60250a" is_pre_election: true
I20260620 18:35:14.757686 24232 leader_election.cc:302] T 00000000000000000000000000000000 P 96a60e8e0ad542b6af066822eb9c6cd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 96a60e8e0ad542b6af066822eb9c6cd0; no voters: 09436352da2444d982c7c3b83b60250a, 3e9d967736254d00b88c77850b39bfb0
I20260620 18:35:14.758030 24458 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 96a60e8e0ad542b6af066822eb9c6cd0 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 18:35:14.758504 24380 leader_election.cc:302] T 00000000000000000000000000000000 P 3e9d967736254d00b88c77850b39bfb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09436352da2444d982c7c3b83b60250a, 3e9d967736254d00b88c77850b39bfb0; no voters: 96a60e8e0ad542b6af066822eb9c6cd0
I20260620 18:35:14.759006 24444 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e9d967736254d00b88c77850b39bfb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:35:14.749413 24472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09436352da2444d982c7c3b83b60250a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a60e8e0ad542b6af066822eb9c6cd0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45923 } } peers { permanent_uuid: "09436352da2444d982c7c3b83b60250a" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 44743 } } peers { permanent_uuid: "3e9d967736254d00b88c77850b39bfb0" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36229 } } }
I20260620 18:35:14.768762 24472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09436352da2444d982c7c3b83b60250a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:35:14.772470 24444 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e9d967736254d00b88c77850b39bfb0 [term 1 LEADER]: Becoming Leader. State: Replica: 3e9d967736254d00b88c77850b39bfb0, State: Running, Role: LEADER
I20260620 18:35:14.773284 24444 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e9d967736254d00b88c77850b39bfb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a60e8e0ad542b6af066822eb9c6cd0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45923 } } peers { permanent_uuid: "09436352da2444d982c7c3b83b60250a" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 44743 } } peers { permanent_uuid: "3e9d967736254d00b88c77850b39bfb0" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36229 } }
I20260620 18:35:14.795059 24475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e9d967736254d00b88c77850b39bfb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e9d967736254d00b88c77850b39bfb0. Latest consensus state: current_term: 1 leader_uuid: "3e9d967736254d00b88c77850b39bfb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a60e8e0ad542b6af066822eb9c6cd0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45923 } } peers { permanent_uuid: "09436352da2444d982c7c3b83b60250a" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 44743 } } peers { permanent_uuid: "3e9d967736254d00b88c77850b39bfb0" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36229 } } }
I20260620 18:35:14.795511 24475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e9d967736254d00b88c77850b39bfb0 [sys.catalog]: This master's current role is: LEADER
W20260620 18:35:14.801007 24485 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 09436352da2444d982c7c3b83b60250a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:35:14.801187 24485 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:35:14.801941 24487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:35:14.804092 24487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:35:14.819139 24329 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 09436352da2444d982c7c3b83b60250a [term 1 FOLLOWER]: Refusing update from remote peer 3e9d967736254d00b88c77850b39bfb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:35:14.820068 24444 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e9d967736254d00b88c77850b39bfb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09436352da2444d982c7c3b83b60250a" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 44743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:35:14.827539 24258 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 96a60e8e0ad542b6af066822eb9c6cd0 [term 1 FOLLOWER]: Refusing update from remote peer 3e9d967736254d00b88c77850b39bfb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:35:14.828418 24444 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e9d967736254d00b88c77850b39bfb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96a60e8e0ad542b6af066822eb9c6cd0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:35:14.864275 24458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96a60e8e0ad542b6af066822eb9c6cd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e9d967736254d00b88c77850b39bfb0. Latest consensus state: current_term: 1 leader_uuid: "3e9d967736254d00b88c77850b39bfb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a60e8e0ad542b6af066822eb9c6cd0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45923 } } peers { permanent_uuid: "09436352da2444d982c7c3b83b60250a" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 44743 } } peers { permanent_uuid: "3e9d967736254d00b88c77850b39bfb0" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36229 } } }
I20260620 18:35:14.864667 24458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96a60e8e0ad542b6af066822eb9c6cd0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:35:14.867975 24488 mvcc.cc:205] Tried to move back new op lower bound from 7298992188688003072 to 7298992188517847040. Current Snapshot: MvccSnapshot[applied={T|T < 7298992188688003072}]
I20260620 18:35:14.871944 24472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09436352da2444d982c7c3b83b60250a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e9d967736254d00b88c77850b39bfb0. Latest consensus state: current_term: 1 leader_uuid: "3e9d967736254d00b88c77850b39bfb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a60e8e0ad542b6af066822eb9c6cd0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45923 } } peers { permanent_uuid: "09436352da2444d982c7c3b83b60250a" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 44743 } } peers { permanent_uuid: "3e9d967736254d00b88c77850b39bfb0" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36229 } } }
I20260620 18:35:14.872287 24472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09436352da2444d982c7c3b83b60250a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:35:14.878018 24489 mvcc.cc:205] Tried to move back new op lower bound from 7298992188688003072 to 7298992188517847040. Current Snapshot: MvccSnapshot[applied={T|T < 7298992188688003072}]
I20260620 18:35:14.879746 24458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96a60e8e0ad542b6af066822eb9c6cd0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e9d967736254d00b88c77850b39bfb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a60e8e0ad542b6af066822eb9c6cd0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45923 } } peers { permanent_uuid: "09436352da2444d982c7c3b83b60250a" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 44743 } } peers { permanent_uuid: "3e9d967736254d00b88c77850b39bfb0" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36229 } } }
I20260620 18:35:14.880008 24458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96a60e8e0ad542b6af066822eb9c6cd0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:35:14.880589 24472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09436352da2444d982c7c3b83b60250a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e9d967736254d00b88c77850b39bfb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a60e8e0ad542b6af066822eb9c6cd0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45923 } } peers { permanent_uuid: "09436352da2444d982c7c3b83b60250a" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 44743 } } peers { permanent_uuid: "3e9d967736254d00b88c77850b39bfb0" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36229 } } }
I20260620 18:35:14.880927 24472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09436352da2444d982c7c3b83b60250a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:35:14.885349 24444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e9d967736254d00b88c77850b39bfb0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e9d967736254d00b88c77850b39bfb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a60e8e0ad542b6af066822eb9c6cd0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45923 } } peers { permanent_uuid: "09436352da2444d982c7c3b83b60250a" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 44743 } } peers { permanent_uuid: "3e9d967736254d00b88c77850b39bfb0" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36229 } } }
I20260620 18:35:14.885699 24444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e9d967736254d00b88c77850b39bfb0 [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:14.885989 24444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e9d967736254d00b88c77850b39bfb0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e9d967736254d00b88c77850b39bfb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a60e8e0ad542b6af066822eb9c6cd0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45923 } } peers { permanent_uuid: "09436352da2444d982c7c3b83b60250a" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 44743 } } peers { permanent_uuid: "3e9d967736254d00b88c77850b39bfb0" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 36229 } } }
I20260620 18:35:14.886251 24444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e9d967736254d00b88c77850b39bfb0 [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:14.886422 24487 catalog_manager.cc:1347] Generated new cluster ID: 04167124dace4414b73c46c5453e0d13
I20260620 18:35:14.886519 24487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:35:14.907923 24487 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:35:14.909821 24487 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:35:14.930027 24487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e9d967736254d00b88c77850b39bfb0: Generated new TSK 0
I20260620 18:35:14.931128 24487 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:35:15.656100 24469 catalog_manager.cc:1259] Loaded cluster ID: 04167124dace4414b73c46c5453e0d13
I20260620 18:35:15.656232 24469 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 96a60e8e0ad542b6af066822eb9c6cd0: loading cluster ID for follower catalog manager: success
I20260620 18:35:15.658725 24469 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 96a60e8e0ad542b6af066822eb9c6cd0: acquiring CA information for follower catalog manager: success
I20260620 18:35:15.660502 24469 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 96a60e8e0ad542b6af066822eb9c6cd0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:35:15.803148 24485 catalog_manager.cc:1259] Loaded cluster ID: 04167124dace4414b73c46c5453e0d13
I20260620 18:35:15.803319 24485 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 09436352da2444d982c7c3b83b60250a: loading cluster ID for follower catalog manager: success
I20260620 18:35:15.806030 24485 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 09436352da2444d982c7c3b83b60250a: acquiring CA information for follower catalog manager: success
I20260620 18:35:15.807767 24485 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 09436352da2444d982c7c3b83b60250a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:35:16.064459 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 24216
I20260620 18:35:16.065766 24278 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:16.175293 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 24216
I20260620 18:35:16.195989 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 24290
I20260620 18:35:16.197163 24352 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:16.301363 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 24290
I20260620 18:35:16.321177 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 24364
I20260620 18:35:16.322309 24426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:16.428387 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 24364
2026-06-20T18:35:16Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4714 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:35:16 dist-test-slave-cbn3 krb5kdc[24504](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:35:16 dist-test-slave-cbn3 krb5kdc[24504](info): set up 2 sockets
Jun 20 18:35:16 dist-test-slave-cbn3 krb5kdc[24504](info): commencing operation
krb5kdc: starting...
W20260620 18:35:17.525615 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:35:17 dist-test-slave-cbn3 krb5kdc[24504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:35:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:35:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.131.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
I20260620 18:35:17.756830 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:38779
--webserver_interface=127.10.131.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:33359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:38779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:18.079562 24517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:18.079917 24517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:18.080086 24517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:18.087275 24517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:35:18.087373 24517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:18.087476 24517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:18.087576 24517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:35:18.087661 24517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:35:18.097589 24517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:33359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:38779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:38779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:35:18.098872 24517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:18.100167 24517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:18.111835 24525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:18.111891 24523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:18.111801 24522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:18.112658 24517 server_base.cc:1034] running on GCE node
I20260620 18:35:18.113420 24517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:18.114704 24517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:18.115924 24517 hybrid_clock.cc:648] HybridClock initialized: now 1781980518115888 us; error 49 us; skew 500 ppm
I20260620 18:35:18.116254 24517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:18 dist-test-slave-cbn3 krb5kdc[24504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980518, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:18.119333 24517 init.cc:377] Logged in from keytab as kudu/127.10.131.62@KRBTEST.COM (short username kudu)
I20260620 18:35:18.120788 24517 webserver.cc:460] Webserver started at http://127.10.131.62:40079/ using document root <none> and password file <none>
I20260620 18:35:18.121459 24517 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:18.121605 24517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:18.121917 24517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:18.124784 24517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "36eb5ae5b4254e46bb72653f0c1a25c9"
format_stamp: "Formatted at 2026-06-20 18:35:18 on dist-test-slave-cbn3"
server_key: "b22d2688dd0233ab2914bf70d324fef2"
server_key_iv: "5076459488c0a5578bea0399f5cee6b2"
server_key_version: "encryptionkey@0"
I20260620 18:35:18.125597 24517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "36eb5ae5b4254e46bb72653f0c1a25c9"
format_stamp: "Formatted at 2026-06-20 18:35:18 on dist-test-slave-cbn3"
server_key: "b22d2688dd0233ab2914bf70d324fef2"
server_key_iv: "5076459488c0a5578bea0399f5cee6b2"
server_key_version: "encryptionkey@0"
I20260620 18:35:18.130605 24517 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:35:18.134032 24532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:18.135442 24517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:35:18.135581 24517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "36eb5ae5b4254e46bb72653f0c1a25c9"
format_stamp: "Formatted at 2026-06-20 18:35:18 on dist-test-slave-cbn3"
server_key: "b22d2688dd0233ab2914bf70d324fef2"
server_key_iv: "5076459488c0a5578bea0399f5cee6b2"
server_key_version: "encryptionkey@0"
I20260620 18:35:18.135751 24517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:18.171993 24517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:18.176339 24517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:18.176575 24517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:18.195641 24517 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:38779
I20260620 18:35:18.195652 24583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:38779 every 8 connection(s)
I20260620 18:35:18.197045 24517 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:35:18.198482 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 24517
I20260620 18:35:18.198732 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:35:18.199393 10764 external_mini_cluster.cc:1428] Setting key 98070ca2f7281981033e955af90ed4d8
I20260620 18:35:18.201274 24584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:35:18 dist-test-slave-cbn3 krb5kdc[24504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:35:18.214856 24584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36eb5ae5b4254e46bb72653f0c1a25c9: Bootstrap starting.
I20260620 18:35:18.219046 24584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36eb5ae5b4254e46bb72653f0c1a25c9: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:18.220368 24584 log.cc:824] T 00000000000000000000000000000000 P 36eb5ae5b4254e46bb72653f0c1a25c9: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:18.223888 24584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36eb5ae5b4254e46bb72653f0c1a25c9: No bootstrap required, opened a new log
I20260620 18:35:18.224617 24587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:18.201298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54292 (local address 127.10.131.62:38779)
0620 18:35:18.202132 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:18.202146 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:18.202196 (+ 50us) server_negotiation.cc:407] Connection header received
0620 18:35:18.204295 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:18.204320 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:18.204804 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:18.205273 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:18.206648 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:18.208737 (+ 2089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:18.210203 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:18.211275 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:18.211480 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:18.215884 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:18.215896 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:18.215915 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:18.215961 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:18.219950 (+ 3989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:18.220728 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:18.220736 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:18.220744 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:18.221002 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:18.221519 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:18.221523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:18.221525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:18.222191 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:18.222498 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:18.222855 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 18:35:18.223301 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":613,"thread_start_us":327,"threads_started":1}
I20260620 18:35:18.228484 10764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:35:18.232627 24584 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36eb5ae5b4254e46bb72653f0c1a25c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36eb5ae5b4254e46bb72653f0c1a25c9" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38779 } }
I20260620 18:35:18.233063 24584 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36eb5ae5b4254e46bb72653f0c1a25c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:18.233181 24584 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36eb5ae5b4254e46bb72653f0c1a25c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36eb5ae5b4254e46bb72653f0c1a25c9, State: Initialized, Role: FOLLOWER
I20260620 18:35:18.233750 24584 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36eb5ae5b4254e46bb72653f0c1a25c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36eb5ae5b4254e46bb72653f0c1a25c9" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38779 } }
I20260620 18:35:18.233986 24584 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36eb5ae5b4254e46bb72653f0c1a25c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:35:18.234109 24584 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36eb5ae5b4254e46bb72653f0c1a25c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:35:18.234284 24584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36eb5ae5b4254e46bb72653f0c1a25c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:18.236402 24584 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36eb5ae5b4254e46bb72653f0c1a25c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36eb5ae5b4254e46bb72653f0c1a25c9" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38779 } }
I20260620 18:35:18.237011 24584 leader_election.cc:302] T 00000000000000000000000000000000 P 36eb5ae5b4254e46bb72653f0c1a25c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36eb5ae5b4254e46bb72653f0c1a25c9; no voters:
I20260620 18:35:18.237566 24584 leader_election.cc:288] T 00000000000000000000000000000000 P 36eb5ae5b4254e46bb72653f0c1a25c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:35:18.237716 24589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36eb5ae5b4254e46bb72653f0c1a25c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:35:18.238380 24589 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36eb5ae5b4254e46bb72653f0c1a25c9 [term 1 LEADER]: Becoming Leader. State: Replica: 36eb5ae5b4254e46bb72653f0c1a25c9, State: Running, Role: LEADER
I20260620 18:35:18.238965 24589 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36eb5ae5b4254e46bb72653f0c1a25c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36eb5ae5b4254e46bb72653f0c1a25c9" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38779 } }
I20260620 18:35:18.239637 24584 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36eb5ae5b4254e46bb72653f0c1a25c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:35:18.241848 24591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36eb5ae5b4254e46bb72653f0c1a25c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36eb5ae5b4254e46bb72653f0c1a25c9. Latest consensus state: current_term: 1 leader_uuid: "36eb5ae5b4254e46bb72653f0c1a25c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36eb5ae5b4254e46bb72653f0c1a25c9" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38779 } } }
I20260620 18:35:18.242245 24591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36eb5ae5b4254e46bb72653f0c1a25c9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:18.242501 24590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36eb5ae5b4254e46bb72653f0c1a25c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36eb5ae5b4254e46bb72653f0c1a25c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36eb5ae5b4254e46bb72653f0c1a25c9" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38779 } } }
I20260620 18:35:18.242867 24590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36eb5ae5b4254e46bb72653f0c1a25c9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:18.244956 24596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:35:18.250149 24596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:35:18.258942 24596 catalog_manager.cc:1347] Generated new cluster ID: b7fe04250a144e95b0be7465fa467509
I20260620 18:35:18.259063 24596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:35:18.268148 24596 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:35:18.269954 24596 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:35:18.282828 24596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36eb5ae5b4254e46bb72653f0c1a25c9: Generated new TSK 0
I20260620 18:35:18.283875 24596 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:35:18.340325 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 24517
I20260620 18:35:18.341481 24579 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:18.436587 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 24517
2026-06-20T18:35:18Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2003 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:35:18 dist-test-slave-cbn3 krb5kdc[24612](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:35:18 dist-test-slave-cbn3 krb5kdc[24612](info): set up 2 sockets
Jun 20 18:35:18 dist-test-slave-cbn3 krb5kdc[24612](info): commencing operation
krb5kdc: starting...
W20260620 18:35:19.537354 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:35:19 dist-test-slave-cbn3 krb5kdc[24612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:35:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:35:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.131.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
I20260620 18:35:19.770181 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:35471
--webserver_interface=127.10.131.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:35471
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:20.096418 24625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:20.096732 24625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:20.096943 24625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:20.104398 24625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:35:20.104511 24625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:20.104560 24625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:20.104642 24625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:35:20.104712 24625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:35:20.115024 24625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:35471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:35471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:35:20.116393 24625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:20.117756 24625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:20.129716 24630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:20.129966 24633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:20.130060 24631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:20.131145 24625 server_base.cc:1034] running on GCE node
I20260620 18:35:20.132083 24625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:20.133658 24625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:20.134919 24625 hybrid_clock.cc:648] HybridClock initialized: now 1781980520134871 us; error 57 us; skew 500 ppm
I20260620 18:35:20.135334 24625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:20 dist-test-slave-cbn3 krb5kdc[24612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980520, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:20.138612 24625 init.cc:377] Logged in from keytab as kudu/127.10.131.62@KRBTEST.COM (short username kudu)
I20260620 18:35:20.140244 24625 webserver.cc:460] Webserver started at http://127.10.131.62:39297/ using document root <none> and password file <none>
I20260620 18:35:20.141036 24625 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:20.141197 24625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:20.141523 24625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:20.144541 24625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "4c29fa9c04464cf3930b9025bcf3fa5b"
format_stamp: "Formatted at 2026-06-20 18:35:20 on dist-test-slave-cbn3"
server_key: "f55fbaf3b5453f685bbb8f648227503b"
server_key_iv: "f5b7a4c0dad3df7f34ed65b7fd071f21"
server_key_version: "encryptionkey@0"
I20260620 18:35:20.145337 24625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "4c29fa9c04464cf3930b9025bcf3fa5b"
format_stamp: "Formatted at 2026-06-20 18:35:20 on dist-test-slave-cbn3"
server_key: "f55fbaf3b5453f685bbb8f648227503b"
server_key_iv: "f5b7a4c0dad3df7f34ed65b7fd071f21"
server_key_version: "encryptionkey@0"
I20260620 18:35:20.150424 24625 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:35:20.153856 24640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:20.155238 24625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:35:20.155422 24625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "4c29fa9c04464cf3930b9025bcf3fa5b"
format_stamp: "Formatted at 2026-06-20 18:35:20 on dist-test-slave-cbn3"
server_key: "f55fbaf3b5453f685bbb8f648227503b"
server_key_iv: "f5b7a4c0dad3df7f34ed65b7fd071f21"
server_key_version: "encryptionkey@0"
I20260620 18:35:20.155613 24625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:20.173830 24625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:20.177812 24625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:20.178120 24625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:20.196560 24625 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:35471
I20260620 18:35:20.196592 24691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:35471 every 8 connection(s)
I20260620 18:35:20.197960 24625 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:35:20.201588 24692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:20.202364 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 24625
I20260620 18:35:20.202574 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:35:20.203437 10764 external_mini_cluster.cc:1428] Setting key df7590d99f6f15427191a54ea80d7a11
I20260620 18:35:20.216046 24692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b: Bootstrap starting.
Jun 20 18:35:20 dist-test-slave-cbn3 krb5kdc[24612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:35:20.220041 24692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:20.221184 24692 log.cc:824] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:20.224424 24692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b: No bootstrap required, opened a new log
I20260620 18:35:20.228312 24695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:20.205582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42524 (local address 127.10.131.62:35471)
0620 18:35:20.206380 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:20.206391 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:20.206431 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:35:20.208104 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:20.208127 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:20.208532 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:20.208969 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:20.210728 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:20.212043 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:20.213958 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:20.214619 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:20.214744 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:20.219871 (+ 5127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:20.219879 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:20.219893 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:20.219929 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:20.223674 (+ 3745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:20.224415 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:20.224425 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:20.224433 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:20.224630 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:20.225083 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:20.225090 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:20.225098 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:20.225813 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:20.226163 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:20.226450 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 18:35:20.226903 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":575,"thread_start_us":392,"threads_started":1}
I20260620 18:35:20.233881 24692 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c29fa9c04464cf3930b9025bcf3fa5b" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35471 } }
I20260620 18:35:20.234205 24692 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:20.234292 24692 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c29fa9c04464cf3930b9025bcf3fa5b, State: Initialized, Role: FOLLOWER
I20260620 18:35:20.234760 24692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c29fa9c04464cf3930b9025bcf3fa5b" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35471 } }
I20260620 18:35:20.234952 24692 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:35:20.235044 24692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:35:20.235152 24692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:20.237469 24692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c29fa9c04464cf3930b9025bcf3fa5b" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35471 } }
I20260620 18:35:20.237915 24692 leader_election.cc:302] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c29fa9c04464cf3930b9025bcf3fa5b; no voters:
I20260620 18:35:20.238415 24692 leader_election.cc:288] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:35:20.238552 24697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:35:20.239290 24697 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [term 1 LEADER]: Becoming Leader. State: Replica: 4c29fa9c04464cf3930b9025bcf3fa5b, State: Running, Role: LEADER
I20260620 18:35:20.239778 24697 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c29fa9c04464cf3930b9025bcf3fa5b" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35471 } }
I20260620 18:35:20.240473 24692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:35:20.242997 24699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c29fa9c04464cf3930b9025bcf3fa5b. Latest consensus state: current_term: 1 leader_uuid: "4c29fa9c04464cf3930b9025bcf3fa5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c29fa9c04464cf3930b9025bcf3fa5b" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35471 } } }
I20260620 18:35:20.243451 24699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:20.244309 24698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c29fa9c04464cf3930b9025bcf3fa5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c29fa9c04464cf3930b9025bcf3fa5b" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35471 } } }
I20260620 18:35:20.244760 24698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:20.245082 24705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:35:20.250183 24705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:35:20.259613 24705 catalog_manager.cc:1347] Generated new cluster ID: 956ff27937fd49cfb92097080434316a
I20260620 18:35:20.259792 24705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:35:20.283174 24705 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:35:20.284502 24705 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:35:20.295140 24705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b: Generated new TSK 0
I20260620 18:35:20.295992 24705 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.131.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab.
Entry for principal kudu/127.10.131.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab.
I20260620 18:35:20.358693 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.1:0
--local_ip_for_outbound_sockets=127.10.131.1
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:35471
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:20.691892 24718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:20.692193 24718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:20.692399 24718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:20.699766 24718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:20.699939 24718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:20.700161 24718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.1
I20260620 18:35:20.710747 24718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:35471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.1
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:35:20.712117 24718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:20.713459 24718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:20.726439 24723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:20.726482 24724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:20.726617 24726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:20.727907 24718 server_base.cc:1034] running on GCE node
I20260620 18:35:20.728502 24718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:20.729451 24718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:20.730647 24718 hybrid_clock.cc:648] HybridClock initialized: now 1781980520730601 us; error 38 us; skew 500 ppm
I20260620 18:35:20.730962 24718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:20 dist-test-slave-cbn3 krb5kdc[24612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980520, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:20.734017 24718 init.cc:377] Logged in from keytab as kudu/127.10.131.1@KRBTEST.COM (short username kudu)
I20260620 18:35:20.735462 24718 webserver.cc:460] Webserver started at http://127.10.131.1:38873/ using document root <none> and password file <none>
I20260620 18:35:20.736147 24718 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:20.736294 24718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:20.736594 24718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:20.739488 24718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/data/instance:
uuid: "f52b8e69f724413281eff5ef630ed0fa"
format_stamp: "Formatted at 2026-06-20 18:35:20 on dist-test-slave-cbn3"
server_key: "4a5d3f454279ccd50a0dca085831dc67"
server_key_iv: "c9432bfe3a5f3766c2f396fa3bc0b5e2"
server_key_version: "encryptionkey@0"
I20260620 18:35:20.740309 24718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance:
uuid: "f52b8e69f724413281eff5ef630ed0fa"
format_stamp: "Formatted at 2026-06-20 18:35:20 on dist-test-slave-cbn3"
server_key: "4a5d3f454279ccd50a0dca085831dc67"
server_key_iv: "c9432bfe3a5f3766c2f396fa3bc0b5e2"
server_key_version: "encryptionkey@0"
I20260620 18:35:20.745544 24718 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:35:20.750497 24733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:20.752316 24718 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 18:35:20.752483 24718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/wal
uuid: "f52b8e69f724413281eff5ef630ed0fa"
format_stamp: "Formatted at 2026-06-20 18:35:20 on dist-test-slave-cbn3"
server_key: "4a5d3f454279ccd50a0dca085831dc67"
server_key_iv: "c9432bfe3a5f3766c2f396fa3bc0b5e2"
server_key_version: "encryptionkey@0"
I20260620 18:35:20.752691 24718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:20.776252 24718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:20.780546 24718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:20.780782 24718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:20.781754 24718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:20.783043 24718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:20.783118 24718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:20.783200 24718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:20.783241 24718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:20.814857 24718 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.1:38513
I20260620 18:35:20.814906 24845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.1:38513 every 8 connection(s)
I20260620 18:35:20.816102 24718 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
I20260620 18:35:20.822705 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 24718
I20260620 18:35:20.822930 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance
I20260620 18:35:20.823678 10764 external_mini_cluster.cc:1428] Setting key 6077156f6853e6ff2027e022721bf64d
Jun 20 18:35:20 dist-test-slave-cbn3 krb5kdc[24612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980520, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:35:20.836643 24848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:20.818601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:43227 (local address 127.10.131.62:35471)
0620 18:35:20.819004 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:20.819009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:20.819955 (+ 946us) server_negotiation.cc:407] Connection header received
0620 18:35:20.821795 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:20.821799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:20.821867 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:20.821962 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:20.823469 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:20.824479 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:20.826455 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:20.827195 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:20.827343 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:20.830596 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:20.830605 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:20.830613 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:20.830656 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:20.833739 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:20.834444 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:20.834449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:20.834453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:20.834574 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:20.834995 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:20.835000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:20.835003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:20.835257 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:20.835414 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:20.836070 (+ 656us) server_negotiation.cc:299] Negotiation successful
0620 18:35:20.836279 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":199,"threads_started":1}
I20260620 18:35:20.837553 24849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:20.819136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.1:43227)
0620 18:35:20.819799 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:20.819836 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:20.821536 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:20.822155 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:20.822163 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:20.822643 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:20.823281 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:20.823299 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:20.824649 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:20.824653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:20.826295 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:20.826306 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:20.827397 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:20.827402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:20.827581 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:20.828241 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:20.828262 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:20.830384 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:20.833886 (+ 3502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:20.833895 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:20.833910 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:20.834318 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:20.834692 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:20.834696 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:20.834701 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:20.834877 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:20.835385 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:20.835393 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:20.835699 (+ 306us) client_negotiation.cc:769] Sending connection context
0620 18:35:20.835955 (+ 256us) client_negotiation.cc:240] Negotiation successful
0620 18:35:20.836465 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":264,"threads_started":1}
I20260620 18:35:20.839004 24846 heartbeater.cc:347] Connected to a master server at 127.10.131.62:35471
I20260620 18:35:20.839295 24846 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:20.839972 24846 heartbeater.cc:511] Master 127.10.131.62:35471 requested a full tablet report, sending...
I20260620 18:35:20.842271 24657 ts_manager.cc:194] Registered new tserver with Master: f52b8e69f724413281eff5ef630ed0fa (127.10.131.1:38513)
I20260620 18:35:20.843971 24657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.1@KRBTEST.COM'} at 127.10.131.1:43227
WARNING: no policy specified for kudu/127.10.131.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab.
Entry for principal kudu/127.10.131.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab.
I20260620 18:35:20.875068 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.2:0
--local_ip_for_outbound_sockets=127.10.131.2
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:35471
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:21.194677 24852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:21.195000 24852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:21.195180 24852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:21.202730 24852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:21.202890 24852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:21.203063 24852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.2
I20260620 18:35:21.212908 24852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:35471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.2
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:35:21.214227 24852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:21.215606 24852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:21.228693 24857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:21.228618 24860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:21.228674 24858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:21.229552 24852 server_base.cc:1034] running on GCE node
I20260620 18:35:21.230137 24852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:21.231081 24852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:21.232304 24852 hybrid_clock.cc:648] HybridClock initialized: now 1781980521232266 us; error 51 us; skew 500 ppm
I20260620 18:35:21.232612 24852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:21 dist-test-slave-cbn3 krb5kdc[24612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980521, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:21.235682 24852 init.cc:377] Logged in from keytab as kudu/127.10.131.2@KRBTEST.COM (short username kudu)
I20260620 18:35:21.237188 24852 webserver.cc:460] Webserver started at http://127.10.131.2:41945/ using document root <none> and password file <none>
I20260620 18:35:21.237843 24852 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:21.237985 24852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:21.238291 24852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:21.241237 24852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/data/instance:
uuid: "ee6278c92e2b4dcdafaa6356dab839f7"
format_stamp: "Formatted at 2026-06-20 18:35:21 on dist-test-slave-cbn3"
server_key: "ed66ecdb9bad87d9d621d1da087b1b8f"
server_key_iv: "a702c8bf2ed5b0ee0bda99be877c0ffc"
server_key_version: "encryptionkey@0"
I20260620 18:35:21.242074 24852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance:
uuid: "ee6278c92e2b4dcdafaa6356dab839f7"
format_stamp: "Formatted at 2026-06-20 18:35:21 on dist-test-slave-cbn3"
server_key: "ed66ecdb9bad87d9d621d1da087b1b8f"
server_key_iv: "a702c8bf2ed5b0ee0bda99be877c0ffc"
server_key_version: "encryptionkey@0"
I20260620 18:35:21.247668 24852 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:35:21.251051 24867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:21.252285 24852 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:35:21.252460 24852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/wal
uuid: "ee6278c92e2b4dcdafaa6356dab839f7"
format_stamp: "Formatted at 2026-06-20 18:35:21 on dist-test-slave-cbn3"
server_key: "ed66ecdb9bad87d9d621d1da087b1b8f"
server_key_iv: "a702c8bf2ed5b0ee0bda99be877c0ffc"
server_key_version: "encryptionkey@0"
I20260620 18:35:21.252656 24852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:21.280791 24852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:21.285013 24852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:21.285343 24852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:21.286499 24852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:21.287999 24852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:21.288110 24852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:21.288273 24852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:21.288347 24852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:21.321137 24852 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.2:42859
I20260620 18:35:21.321189 24979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.2:42859 every 8 connection(s)
I20260620 18:35:21.322428 24852 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
I20260620 18:35:21.326749 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 24852
I20260620 18:35:21.326938 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance
I20260620 18:35:21.327544 10764 external_mini_cluster.cc:1428] Setting key c74cc6f1b187adf3fc0bfbf0225131a5
Jun 20 18:35:21 dist-test-slave-cbn3 krb5kdc[24612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980521, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:35:21.344527 24848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:21.324958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:59413 (local address 127.10.131.62:35471)
0620 18:35:21.325146 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:21.325151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:21.326349 (+ 1198us) server_negotiation.cc:407] Connection header received
0620 18:35:21.328731 (+ 2382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:21.328735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:21.328803 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:21.328946 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:21.330775 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:21.331451 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:21.333941 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:21.334355 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:21.334450 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:21.338605 (+ 4155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:21.338610 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:21.338613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:21.338644 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:21.341497 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:21.342293 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:21.342302 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:21.342304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:21.342425 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:21.342842 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:21.342846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:21.342851 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:21.343102 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:21.343253 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:21.343926 (+ 673us) server_negotiation.cc:299] Negotiation successful
0620 18:35:21.344159 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:35:21.345616 24982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:21.325414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.2:59413)
0620 18:35:21.326183 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:21.326228 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:21.328437 (+ 2209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:21.329174 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:21.329185 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:21.329820 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:21.330584 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:21.330626 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:21.331611 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:21.331619 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:21.333800 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:21.333819 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:21.334584 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:21.334603 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:21.334935 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:21.335682 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:21.335714 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:21.338359 (+ 2645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:21.341657 (+ 3298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:21.341666 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:21.341682 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:21.342168 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:21.342533 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:21.342539 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:21.342545 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:21.342723 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:21.343224 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:21.343232 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:21.343581 (+ 349us) client_negotiation.cc:769] Sending connection context
0620 18:35:21.343831 (+ 250us) client_negotiation.cc:240] Negotiation successful
0620 18:35:21.344380 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":299,"threads_started":1}
I20260620 18:35:21.347201 24980 heartbeater.cc:347] Connected to a master server at 127.10.131.62:35471
I20260620 18:35:21.347460 24980 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:21.348187 24980 heartbeater.cc:511] Master 127.10.131.62:35471 requested a full tablet report, sending...
I20260620 18:35:21.349762 24657 ts_manager.cc:194] Registered new tserver with Master: ee6278c92e2b4dcdafaa6356dab839f7 (127.10.131.2:42859)
I20260620 18:35:21.351037 24657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.2@KRBTEST.COM'} at 127.10.131.2:59413
WARNING: no policy specified for kudu/127.10.131.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab.
Entry for principal kudu/127.10.131.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab.
I20260620 18:35:21.382170 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.3:0
--local_ip_for_outbound_sockets=127.10.131.3
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:35471
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:21.704102 24985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:21.704427 24985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:21.704591 24985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:21.711864 24985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:21.712016 24985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:21.712200 24985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.3
I20260620 18:35:21.722011 24985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:35471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.3
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:35:21.723331 24985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:21.724599 24985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:21.738184 24991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:21.738224 24990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:21.738231 24993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:21.739123 24985 server_base.cc:1034] running on GCE node
I20260620 18:35:21.739719 24985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:21.740604 24985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:21.741851 24985 hybrid_clock.cc:648] HybridClock initialized: now 1781980521741763 us; error 51 us; skew 500 ppm
I20260620 18:35:21.742220 24985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:21 dist-test-slave-cbn3 krb5kdc[24612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980521, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:21.745517 24985 init.cc:377] Logged in from keytab as kudu/127.10.131.3@KRBTEST.COM (short username kudu)
I20260620 18:35:21.747126 24985 webserver.cc:460] Webserver started at http://127.10.131.3:32829/ using document root <none> and password file <none>
I20260620 18:35:21.747769 24985 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:21.747872 24985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:21.748143 24985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:21.751039 24985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/data/instance:
uuid: "4696a52c99814944b06cc2a4542d5584"
format_stamp: "Formatted at 2026-06-20 18:35:21 on dist-test-slave-cbn3"
server_key: "4d8aa051697d1eed27322cdb0dd34ffa"
server_key_iv: "f78c6758c8c91cbac28595683e62aaeb"
server_key_version: "encryptionkey@0"
I20260620 18:35:21.751767 24985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance:
uuid: "4696a52c99814944b06cc2a4542d5584"
format_stamp: "Formatted at 2026-06-20 18:35:21 on dist-test-slave-cbn3"
server_key: "4d8aa051697d1eed27322cdb0dd34ffa"
server_key_iv: "f78c6758c8c91cbac28595683e62aaeb"
server_key_version: "encryptionkey@0"
I20260620 18:35:21.756827 24985 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 18:35:21.760496 25000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:21.761788 24985 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:35:21.761916 24985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/wal
uuid: "4696a52c99814944b06cc2a4542d5584"
format_stamp: "Formatted at 2026-06-20 18:35:21 on dist-test-slave-cbn3"
server_key: "4d8aa051697d1eed27322cdb0dd34ffa"
server_key_iv: "f78c6758c8c91cbac28595683e62aaeb"
server_key_version: "encryptionkey@0"
I20260620 18:35:21.762065 24985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:21.780705 24985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:21.785197 24985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:21.785441 24985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:21.786337 24985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:21.787588 24985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:21.787660 24985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:21.787745 24985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:21.787787 24985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:21.819825 24985 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.3:45173
I20260620 18:35:21.819844 25112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.3:45173 every 8 connection(s)
I20260620 18:35:21.821349 24985 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
I20260620 18:35:21.823935 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 24985
I20260620 18:35:21.824119 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance
I20260620 18:35:21.824591 10764 external_mini_cluster.cc:1428] Setting key 67a08a7b435734c70d1806f127f965d0
Jun 20 18:35:21 dist-test-slave-cbn3 krb5kdc[24612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980521, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:35:21.843708 24848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:21.823950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:49431 (local address 127.10.131.62:35471)
0620 18:35:21.824157 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:21.824165 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:21.825420 (+ 1255us) server_negotiation.cc:407] Connection header received
0620 18:35:21.827181 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:21.827186 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:21.827280 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:21.827394 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:21.828830 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:21.829608 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:21.831061 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:21.831477 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:21.831606 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:21.837692 (+ 6086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:21.837699 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:21.837704 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:21.837745 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:21.841030 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:21.841750 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:21.841754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:21.841756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:21.841882 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:21.842282 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:21.842285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:21.842287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:21.842554 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:21.842678 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:21.843299 (+ 621us) server_negotiation.cc:299] Negotiation successful
0620 18:35:21.843462 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 18:35:21.844622 25115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:21.824430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.3:49431)
0620 18:35:21.825262 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:21.825313 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:21.826926 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:21.827630 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:21.827638 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:21.828147 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:21.828667 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:21.828685 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:21.829764 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:21.829768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:21.830921 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:21.830933 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:21.831709 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:21.831716 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:21.831945 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:21.835292 (+ 3347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:21.835321 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:21.837443 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:21.841172 (+ 3729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:21.841178 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:21.841191 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:21.841655 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:21.841971 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:21.841974 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:21.841976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:21.842193 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:21.842652 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:21.842676 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:21.842997 (+ 321us) client_negotiation.cc:769] Sending connection context
0620 18:35:21.843222 (+ 225us) client_negotiation.cc:240] Negotiation successful
0620 18:35:21.843594 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":375,"threads_started":1}
I20260620 18:35:21.846102 25113 heartbeater.cc:347] Connected to a master server at 127.10.131.62:35471
I20260620 18:35:21.846459 25113 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:21.847237 25113 heartbeater.cc:511] Master 127.10.131.62:35471 requested a full tablet report, sending...
I20260620 18:35:21.847465 24846 heartbeater.cc:503] Master 127.10.131.62:35471 was elected leader, sending a full tablet report...
I20260620 18:35:21.848829 24656 ts_manager.cc:194] Registered new tserver with Master: 4696a52c99814944b06cc2a4542d5584 (127.10.131.3:45173)
I20260620 18:35:21.849661 24656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.3@KRBTEST.COM'} at 127.10.131.3:49431
I20260620 18:35:21.852152 10764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:35:21.868702 24848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:21.856521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42538 (local address 127.10.131.62:35471)
0620 18:35:21.856710 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:21.856715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:21.857261 (+ 546us) server_negotiation.cc:407] Connection header received
0620 18:35:21.857448 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:21.857451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:21.857509 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:21.857582 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:21.859043 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:21.859804 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:21.861142 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:21.861521 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:21.861614 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:21.863749 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:21.863753 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:21.863756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:21.863793 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:21.866385 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:21.867112 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:21.867115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:21.867117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:21.867241 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:21.867552 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:21.867555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:21.867557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:21.867870 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:21.868066 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:21.868321 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 18:35:21.868475 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 18:35:21.872695 24656 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42538:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:35:21.879451 24656 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:35:21.907159 25126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:21.891549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55372 (local address 127.10.131.3:45173)
0620 18:35:21.898894 (+ 7345us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:21.898900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:21.898927 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:35:21.899008 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:21.899013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:21.899166 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:21.899376 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:21.900110 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:21.901624 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:21.904583 (+ 2959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:21.906006 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:21.906148 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:21.906381 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:21.906643 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 18:35:21.906823 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7179,"thread_start_us":7043,"threads_started":1}
I20260620 18:35:21.907222 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:21.893279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53694 (local address 127.10.131.2:42859)
0620 18:35:21.893845 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:21.893854 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:21.898492 (+ 4638us) server_negotiation.cc:407] Connection header received
0620 18:35:21.898746 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:21.898754 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:21.898937 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:21.899095 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:21.900427 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:21.902078 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:21.904707 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:21.906124 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:21.906222 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:21.906373 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:21.906734 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 18:35:21.906896 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":754,"threads_started":1}
I20260620 18:35:21.907245 25125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:21.891384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:45173 (local address 127.0.0.1:55372)
0620 18:35:21.892319 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:21.892361 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:21.892517 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:21.899394 (+ 6877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:21.899402 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:21.899467 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:21.899905 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:21.899920 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:21.901755 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:21.901763 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:21.904206 (+ 2443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:21.904220 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:21.906211 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:21.906217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:21.906424 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:21.906461 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 18:35:21.906599 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 18:35:21.906795 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"mutex_wait_us":5,"spinlock_wait_cycles":34048,"thread_start_us":593,"threads_started":1}
I20260620 18:35:21.907537 25128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:21.892503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56020 (local address 127.10.131.1:38513)
0620 18:35:21.893252 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:21.893261 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:21.898259 (+ 4998us) server_negotiation.cc:407] Connection header received
0620 18:35:21.898605 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:21.898617 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:21.898910 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:21.899097 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:21.900438 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:21.902078 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:21.904070 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:21.905540 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:21.905691 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:21.905884 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:21.907023 (+ 1139us) server_negotiation.cc:299] Negotiation successful
0620 18:35:21.907185 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":464,"threads_started":1}
I20260620 18:35:21.907757 25127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:21.892733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:42859 (local address 127.0.0.1:53694)
0620 18:35:21.898390 (+ 5657us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:21.898404 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:21.898585 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:21.899399 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:21.899405 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:21.899488 (+ 83us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:21.900196 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:21.900214 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:21.902224 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:21.902231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:21.904531 (+ 2300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:21.904545 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:21.906418 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:21.906423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:21.906571 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:21.906608 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 18:35:21.906682 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 18:35:21.906795 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5546,"mutex_wait_us":68,"spinlock_wait_cycles":29184,"thread_start_us":5061,"threads_started":1}
I20260620 18:35:21.908638 25129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:21.891738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:38513 (local address 127.0.0.1:56020)
0620 18:35:21.898122 (+ 6384us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:21.898144 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:21.898382 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:21.899399 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:21.899405 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:21.899495 (+ 90us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:21.900229 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:21.900241 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:21.902271 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:21.902275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:21.903906 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:21.903919 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:21.905985 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:21.905990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:21.906190 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:21.906853 (+ 663us) client_negotiation.cc:769] Sending connection context
0620 18:35:21.908151 (+ 1298us) client_negotiation.cc:240] Negotiation successful
0620 18:35:21.908304 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6208,"mutex_wait_us":723,"spinlock_wait_cycles":93824,"thread_start_us":6394,"threads_started":1}
I20260620 18:35:21.911296 24915 tablet_service.cc:1467] Processing CreateTablet for tablet 31bdb5076e0e40c79ed8564c0e4bb215 (DEFAULT_TABLE table=test-table [id=d7ae50109dd84b51a3848654c3124422]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:21.911526 25048 tablet_service.cc:1467] Processing CreateTablet for tablet 31bdb5076e0e40c79ed8564c0e4bb215 (DEFAULT_TABLE table=test-table [id=d7ae50109dd84b51a3848654c3124422]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:21.911897 24781 tablet_service.cc:1467] Processing CreateTablet for tablet 31bdb5076e0e40c79ed8564c0e4bb215 (DEFAULT_TABLE table=test-table [id=d7ae50109dd84b51a3848654c3124422]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:21.912379 24915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31bdb5076e0e40c79ed8564c0e4bb215. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:21.912881 25048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31bdb5076e0e40c79ed8564c0e4bb215. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:21.913157 24781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31bdb5076e0e40c79ed8564c0e4bb215. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:21.923930 25131 tablet_bootstrap.cc:492] T 31bdb5076e0e40c79ed8564c0e4bb215 P ee6278c92e2b4dcdafaa6356dab839f7: Bootstrap starting.
I20260620 18:35:21.927127 25132 tablet_bootstrap.cc:492] T 31bdb5076e0e40c79ed8564c0e4bb215 P 4696a52c99814944b06cc2a4542d5584: Bootstrap starting.
I20260620 18:35:21.927444 25131 tablet_bootstrap.cc:654] T 31bdb5076e0e40c79ed8564c0e4bb215 P ee6278c92e2b4dcdafaa6356dab839f7: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:21.928867 25131 log.cc:824] T 31bdb5076e0e40c79ed8564c0e4bb215 P ee6278c92e2b4dcdafaa6356dab839f7: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:21.930265 25133 tablet_bootstrap.cc:492] T 31bdb5076e0e40c79ed8564c0e4bb215 P f52b8e69f724413281eff5ef630ed0fa: Bootstrap starting.
I20260620 18:35:21.930764 25132 tablet_bootstrap.cc:654] T 31bdb5076e0e40c79ed8564c0e4bb215 P 4696a52c99814944b06cc2a4542d5584: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:21.932077 25131 tablet_bootstrap.cc:492] T 31bdb5076e0e40c79ed8564c0e4bb215 P ee6278c92e2b4dcdafaa6356dab839f7: No bootstrap required, opened a new log
I20260620 18:35:21.932057 25132 log.cc:824] T 31bdb5076e0e40c79ed8564c0e4bb215 P 4696a52c99814944b06cc2a4542d5584: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:21.932349 25131 ts_tablet_manager.cc:1397] T 31bdb5076e0e40c79ed8564c0e4bb215 P ee6278c92e2b4dcdafaa6356dab839f7: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 18:35:21.934636 25133 tablet_bootstrap.cc:654] T 31bdb5076e0e40c79ed8564c0e4bb215 P f52b8e69f724413281eff5ef630ed0fa: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:21.936209 25133 log.cc:824] T 31bdb5076e0e40c79ed8564c0e4bb215 P f52b8e69f724413281eff5ef630ed0fa: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:21.939296 25133 tablet_bootstrap.cc:492] T 31bdb5076e0e40c79ed8564c0e4bb215 P f52b8e69f724413281eff5ef630ed0fa: No bootstrap required, opened a new log
I20260620 18:35:21.939528 25133 ts_tablet_manager.cc:1397] T 31bdb5076e0e40c79ed8564c0e4bb215 P f52b8e69f724413281eff5ef630ed0fa: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260620 18:35:21.939851 25132 tablet_bootstrap.cc:492] T 31bdb5076e0e40c79ed8564c0e4bb215 P 4696a52c99814944b06cc2a4542d5584: No bootstrap required, opened a new log
I20260620 18:35:21.940187 25132 ts_tablet_manager.cc:1397] T 31bdb5076e0e40c79ed8564c0e4bb215 P 4696a52c99814944b06cc2a4542d5584: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.009s
I20260620 18:35:21.941395 25131 raft_consensus.cc:348] T 31bdb5076e0e40c79ed8564c0e4bb215 P ee6278c92e2b4dcdafaa6356dab839f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4696a52c99814944b06cc2a4542d5584" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 45173 } } peers { permanent_uuid: "f52b8e69f724413281eff5ef630ed0fa" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38513 } } peers { permanent_uuid: "ee6278c92e2b4dcdafaa6356dab839f7" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 42859 } }
I20260620 18:35:21.941870 25131 raft_consensus.cc:374] T 31bdb5076e0e40c79ed8564c0e4bb215 P ee6278c92e2b4dcdafaa6356dab839f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:21.941977 25131 raft_consensus.cc:729] T 31bdb5076e0e40c79ed8564c0e4bb215 P ee6278c92e2b4dcdafaa6356dab839f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee6278c92e2b4dcdafaa6356dab839f7, State: Initialized, Role: FOLLOWER
I20260620 18:35:21.942627 25131 consensus_queue.cc:260] T 31bdb5076e0e40c79ed8564c0e4bb215 P ee6278c92e2b4dcdafaa6356dab839f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4696a52c99814944b06cc2a4542d5584" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 45173 } } peers { permanent_uuid: "f52b8e69f724413281eff5ef630ed0fa" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38513 } } peers { permanent_uuid: "ee6278c92e2b4dcdafaa6356dab839f7" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 42859 } }
I20260620 18:35:21.943804 24980 heartbeater.cc:503] Master 127.10.131.62:35471 was elected leader, sending a full tablet report...
I20260620 18:35:21.944540 25131 ts_tablet_manager.cc:1428] T 31bdb5076e0e40c79ed8564c0e4bb215 P ee6278c92e2b4dcdafaa6356dab839f7: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260620 18:35:21.949035 25133 raft_consensus.cc:348] T 31bdb5076e0e40c79ed8564c0e4bb215 P f52b8e69f724413281eff5ef630ed0fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4696a52c99814944b06cc2a4542d5584" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 45173 } } peers { permanent_uuid: "f52b8e69f724413281eff5ef630ed0fa" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38513 } } peers { permanent_uuid: "ee6278c92e2b4dcdafaa6356dab839f7" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 42859 } }
I20260620 18:35:21.949489 25133 raft_consensus.cc:374] T 31bdb5076e0e40c79ed8564c0e4bb215 P f52b8e69f724413281eff5ef630ed0fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:21.949612 25133 raft_consensus.cc:729] T 31bdb5076e0e40c79ed8564c0e4bb215 P f52b8e69f724413281eff5ef630ed0fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f52b8e69f724413281eff5ef630ed0fa, State: Initialized, Role: FOLLOWER
I20260620 18:35:21.950261 25133 consensus_queue.cc:260] T 31bdb5076e0e40c79ed8564c0e4bb215 P f52b8e69f724413281eff5ef630ed0fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4696a52c99814944b06cc2a4542d5584" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 45173 } } peers { permanent_uuid: "f52b8e69f724413281eff5ef630ed0fa" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38513 } } peers { permanent_uuid: "ee6278c92e2b4dcdafaa6356dab839f7" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 42859 } }
I20260620 18:35:21.951032 25132 raft_consensus.cc:348] T 31bdb5076e0e40c79ed8564c0e4bb215 P 4696a52c99814944b06cc2a4542d5584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4696a52c99814944b06cc2a4542d5584" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 45173 } } peers { permanent_uuid: "f52b8e69f724413281eff5ef630ed0fa" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38513 } } peers { permanent_uuid: "ee6278c92e2b4dcdafaa6356dab839f7" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 42859 } }
I20260620 18:35:21.951411 25132 raft_consensus.cc:374] T 31bdb5076e0e40c79ed8564c0e4bb215 P 4696a52c99814944b06cc2a4542d5584 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:21.951516 25132 raft_consensus.cc:729] T 31bdb5076e0e40c79ed8564c0e4bb215 P 4696a52c99814944b06cc2a4542d5584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4696a52c99814944b06cc2a4542d5584, State: Initialized, Role: FOLLOWER
I20260620 18:35:21.952102 25133 ts_tablet_manager.cc:1428] T 31bdb5076e0e40c79ed8564c0e4bb215 P f52b8e69f724413281eff5ef630ed0fa: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260620 18:35:21.952101 25132 consensus_queue.cc:260] T 31bdb5076e0e40c79ed8564c0e4bb215 P 4696a52c99814944b06cc2a4542d5584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4696a52c99814944b06cc2a4542d5584" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 45173 } } peers { permanent_uuid: "f52b8e69f724413281eff5ef630ed0fa" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38513 } } peers { permanent_uuid: "ee6278c92e2b4dcdafaa6356dab839f7" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 42859 } }
I20260620 18:35:21.953205 25113 heartbeater.cc:503] Master 127.10.131.62:35471 was elected leader, sending a full tablet report...
I20260620 18:35:21.953625 25132 ts_tablet_manager.cc:1428] T 31bdb5076e0e40c79ed8564c0e4bb215 P 4696a52c99814944b06cc2a4542d5584: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260620 18:35:22.046263 25137 raft_consensus.cc:482] T 31bdb5076e0e40c79ed8564c0e4bb215 P ee6278c92e2b4dcdafaa6356dab839f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:35:22.046633 25137 raft_consensus.cc:504] T 31bdb5076e0e40c79ed8564c0e4bb215 P ee6278c92e2b4dcdafaa6356dab839f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4696a52c99814944b06cc2a4542d5584" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 45173 } } peers { permanent_uuid: "f52b8e69f724413281eff5ef630ed0fa" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38513 } } peers { permanent_uuid: "ee6278c92e2b4dcdafaa6356dab839f7" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 42859 } }
I20260620 18:35:22.048314 25137 leader_election.cc:288] T 31bdb5076e0e40c79ed8564c0e4bb215 P ee6278c92e2b4dcdafaa6356dab839f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4696a52c99814944b06cc2a4542d5584 (127.10.131.3:45173), f52b8e69f724413281eff5ef630ed0fa (127.10.131.1:38513)
I20260620 18:35:22.057538 25128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:22.048800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:34501 (local address 127.10.131.1:38513)
0620 18:35:22.049008 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:22.049015 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:22.049860 (+ 845us) server_negotiation.cc:407] Connection header received
0620 18:35:22.050008 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:22.050013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:22.050081 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:22.050169 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:22.050995 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:22.052098 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:22.055079 (+ 2981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:22.056327 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:22.056482 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:22.056556 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:22.057155 (+ 599us) server_negotiation.cc:299] Negotiation successful
0620 18:35:22.057301 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 18:35:22.057577 25126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:22.048780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:47517 (local address 127.10.131.3:45173)
0620 18:35:22.048993 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:22.049000 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:22.049785 (+ 785us) server_negotiation.cc:407] Connection header received
0620 18:35:22.049983 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:22.049988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:22.050067 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:22.050170 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:22.050945 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:22.052099 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:22.055073 (+ 2974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:22.056337 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:22.056486 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:22.056556 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:22.057236 (+ 680us) server_negotiation.cc:299] Negotiation successful
0620 18:35:22.057392 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 18:35:22.057636 25141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:22.048718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:45173 (local address 127.10.131.2:47517)
0620 18:35:22.049668 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:22.049689 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:22.049841 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:22.050199 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:22.050204 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:22.050306 (+ 102us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:22.050752 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:22.050770 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:22.052235 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:22.052239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:22.054901 (+ 2662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:22.054916 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:22.056602 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:22.056683 (+ 81us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:22.056884 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:22.056980 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 18:35:22.057073 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 18:35:22.057204 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"mutex_wait_us":198,"spinlock_wait_cycles":20736,"thread_start_us":474,"threads_started":1}
I20260620 18:35:22.058161 25140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:22.048718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:38513 (local address 127.10.131.2:34501)
0620 18:35:22.049765 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:22.049788 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:22.049883 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:22.050199 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:22.050204 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:22.050231 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:22.050841 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:22.050855 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:22.052233 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:22.052238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:22.054901 (+ 2663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:22.054916 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:22.056629 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:22.056633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:22.056882 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:22.056981 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 18:35:22.057724 (+ 743us) client_negotiation.cc:240] Negotiation successful
0620 18:35:22.057843 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":932,"mutex_wait_us":284,"spinlock_wait_cycles":16768,"thread_start_us":453,"threads_started":1}
I20260620 18:35:22.059165 25068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31bdb5076e0e40c79ed8564c0e4bb215" candidate_uuid: "ee6278c92e2b4dcdafaa6356dab839f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4696a52c99814944b06cc2a4542d5584" is_pre_election: true
I20260620 18:35:22.059333 24801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31bdb5076e0e40c79ed8564c0e4bb215" candidate_uuid: "ee6278c92e2b4dcdafaa6356dab839f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f52b8e69f724413281eff5ef630ed0fa" is_pre_election: true
I20260620 18:35:22.059808 25068 raft_consensus.cc:2454] T 31bdb5076e0e40c79ed8564c0e4bb215 P 4696a52c99814944b06cc2a4542d5584 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee6278c92e2b4dcdafaa6356dab839f7 in term 0.
I20260620 18:35:22.059829 24801 raft_consensus.cc:2454] T 31bdb5076e0e40c79ed8564c0e4bb215 P f52b8e69f724413281eff5ef630ed0fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee6278c92e2b4dcdafaa6356dab839f7 in term 0.
I20260620 18:35:22.060532 24870 leader_election.cc:302] T 31bdb5076e0e40c79ed8564c0e4bb215 P ee6278c92e2b4dcdafaa6356dab839f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4696a52c99814944b06cc2a4542d5584, ee6278c92e2b4dcdafaa6356dab839f7; no voters:
I20260620 18:35:22.060951 25137 raft_consensus.cc:2781] T 31bdb5076e0e40c79ed8564c0e4bb215 P ee6278c92e2b4dcdafaa6356dab839f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:35:22.061167 25137 raft_consensus.cc:482] T 31bdb5076e0e40c79ed8564c0e4bb215 P ee6278c92e2b4dcdafaa6356dab839f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:35:22.061287 25137 raft_consensus.cc:3037] T 31bdb5076e0e40c79ed8564c0e4bb215 P ee6278c92e2b4dcdafaa6356dab839f7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:22.063551 25137 raft_consensus.cc:504] T 31bdb5076e0e40c79ed8564c0e4bb215 P ee6278c92e2b4dcdafaa6356dab839f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4696a52c99814944b06cc2a4542d5584" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 45173 } } peers { permanent_uuid: "f52b8e69f724413281eff5ef630ed0fa" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38513 } } peers { permanent_uuid: "ee6278c92e2b4dcdafaa6356dab839f7" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 42859 } }
I20260620 18:35:22.064301 25137 leader_election.cc:288] T 31bdb5076e0e40c79ed8564c0e4bb215 P ee6278c92e2b4dcdafaa6356dab839f7 [CANDIDATE]: Term 1 election: Requested vote from peers 4696a52c99814944b06cc2a4542d5584 (127.10.131.3:45173), f52b8e69f724413281eff5ef630ed0fa (127.10.131.1:38513)
I20260620 18:35:22.064674 25068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31bdb5076e0e40c79ed8564c0e4bb215" candidate_uuid: "ee6278c92e2b4dcdafaa6356dab839f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4696a52c99814944b06cc2a4542d5584"
I20260620 18:35:22.064774 24801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31bdb5076e0e40c79ed8564c0e4bb215" candidate_uuid: "ee6278c92e2b4dcdafaa6356dab839f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f52b8e69f724413281eff5ef630ed0fa"
I20260620 18:35:22.064913 25068 raft_consensus.cc:3037] T 31bdb5076e0e40c79ed8564c0e4bb215 P 4696a52c99814944b06cc2a4542d5584 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:22.065026 24801 raft_consensus.cc:3037] T 31bdb5076e0e40c79ed8564c0e4bb215 P f52b8e69f724413281eff5ef630ed0fa [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:22.067350 24801 raft_consensus.cc:2454] T 31bdb5076e0e40c79ed8564c0e4bb215 P f52b8e69f724413281eff5ef630ed0fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee6278c92e2b4dcdafaa6356dab839f7 in term 1.
I20260620 18:35:22.067548 25068 raft_consensus.cc:2454] T 31bdb5076e0e40c79ed8564c0e4bb215 P 4696a52c99814944b06cc2a4542d5584 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee6278c92e2b4dcdafaa6356dab839f7 in term 1.
I20260620 18:35:22.067800 24868 leader_election.cc:302] T 31bdb5076e0e40c79ed8564c0e4bb215 P ee6278c92e2b4dcdafaa6356dab839f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee6278c92e2b4dcdafaa6356dab839f7, f52b8e69f724413281eff5ef630ed0fa; no voters:
I20260620 18:35:22.068142 25137 raft_consensus.cc:2781] T 31bdb5076e0e40c79ed8564c0e4bb215 P ee6278c92e2b4dcdafaa6356dab839f7 [term 1 FOLLOWER]: Leader election won for term 1
W20260620 18:35:22.068528 24847 tablet.cc:2369] T 31bdb5076e0e40c79ed8564c0e4bb215 P f52b8e69f724413281eff5ef630ed0fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:35:22.068632 25137 raft_consensus.cc:686] T 31bdb5076e0e40c79ed8564c0e4bb215 P ee6278c92e2b4dcdafaa6356dab839f7 [term 1 LEADER]: Becoming Leader. State: Replica: ee6278c92e2b4dcdafaa6356dab839f7, State: Running, Role: LEADER
I20260620 18:35:22.069205 25137 consensus_queue.cc:237] T 31bdb5076e0e40c79ed8564c0e4bb215 P ee6278c92e2b4dcdafaa6356dab839f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4696a52c99814944b06cc2a4542d5584" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 45173 } } peers { permanent_uuid: "f52b8e69f724413281eff5ef630ed0fa" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38513 } } peers { permanent_uuid: "ee6278c92e2b4dcdafaa6356dab839f7" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 42859 } }
W20260620 18:35:22.072965 25114 tablet.cc:2369] T 31bdb5076e0e40c79ed8564c0e4bb215 P 4696a52c99814944b06cc2a4542d5584: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:35:22.073647 24656 catalog_manager.cc:5495] T 31bdb5076e0e40c79ed8564c0e4bb215 P ee6278c92e2b4dcdafaa6356dab839f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee6278c92e2b4dcdafaa6356dab839f7 (127.10.131.2). New cstate: current_term: 1 leader_uuid: "ee6278c92e2b4dcdafaa6356dab839f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4696a52c99814944b06cc2a4542d5584" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 45173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f52b8e69f724413281eff5ef630ed0fa" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee6278c92e2b4dcdafaa6356dab839f7" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 42859 } health_report { overall_health: HEALTHY } } }
W20260620 18:35:22.074231 24981 tablet.cc:2369] T 31bdb5076e0e40c79ed8564c0e4bb215 P ee6278c92e2b4dcdafaa6356dab839f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:35:22.094928 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:22.090015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53696 (local address 127.10.131.2:42859)
0620 18:35:22.090245 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:22.090249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:22.090266 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:35:22.090433 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:22.090436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:22.090513 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:22.090627 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:35:22.090945 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:22.091631 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:22.092997 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:22.093381 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:22.093464 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:22.093878 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:35:22.094388 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:35:22.094492 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:22.094562 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 18:35:22.094623 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 18:35:22.102912 24801 raft_consensus.cc:1261] T 31bdb5076e0e40c79ed8564c0e4bb215 P f52b8e69f724413281eff5ef630ed0fa [term 1 FOLLOWER]: Refusing update from remote peer ee6278c92e2b4dcdafaa6356dab839f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:35:22.102912 25068 raft_consensus.cc:1261] T 31bdb5076e0e40c79ed8564c0e4bb215 P 4696a52c99814944b06cc2a4542d5584 [term 1 FOLLOWER]: Refusing update from remote peer ee6278c92e2b4dcdafaa6356dab839f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:35:22.104110 25137 consensus_queue.cc:1035] T 31bdb5076e0e40c79ed8564c0e4bb215 P ee6278c92e2b4dcdafaa6356dab839f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4696a52c99814944b06cc2a4542d5584" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 45173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:35:22.104416 25142 consensus_queue.cc:1035] T 31bdb5076e0e40c79ed8564c0e4bb215 P ee6278c92e2b4dcdafaa6356dab839f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f52b8e69f724413281eff5ef630ed0fa" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:35:22.116899 25145 mvcc.cc:205] Tried to move back new op lower bound from 7298992218523787264 to 7298992218399891456. Current Snapshot: MvccSnapshot[applied={T|T < 7298992218523787264}]
I20260620 18:35:22.120703 25146 mvcc.cc:205] Tried to move back new op lower bound from 7298992218523787264 to 7298992218399891456. Current Snapshot: MvccSnapshot[applied={T|T < 7298992218523787264}]
I20260620 18:35:22.130754 24656 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:35:22.131000 24656 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:35:22.134348 24656 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b: Sending DeleteTablet for 3 replicas of tablet 31bdb5076e0e40c79ed8564c0e4bb215
I20260620 18:35:22.135452 24781 tablet_service.cc:1514] Processing DeleteTablet for tablet 31bdb5076e0e40c79ed8564c0e4bb215 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:35:22 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:56020
I20260620 18:35:22.135506 25048 tablet_service.cc:1514] Processing DeleteTablet for tablet 31bdb5076e0e40c79ed8564c0e4bb215 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:35:22 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:55372
I20260620 18:35:22.135739 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 24718
I20260620 18:35:22.136384 24915 tablet_service.cc:1514] Processing DeleteTablet for tablet 31bdb5076e0e40c79ed8564c0e4bb215 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:35:22 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:53694
I20260620 18:35:22.160252 25158 tablet_replica.cc:330] T 31bdb5076e0e40c79ed8564c0e4bb215 P ee6278c92e2b4dcdafaa6356dab839f7: stopping tablet replica
I20260620 18:35:22.160987 25158 raft_consensus.cc:2229] T 31bdb5076e0e40c79ed8564c0e4bb215 P ee6278c92e2b4dcdafaa6356dab839f7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:35:22.161824 25158 raft_consensus.cc:2258] T 31bdb5076e0e40c79ed8564c0e4bb215 P ee6278c92e2b4dcdafaa6356dab839f7 [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 20 18:35:22 dist-test-slave-cbn3 krb5kdc[24612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.1@KRBTEST.COM
I20260620 18:35:22.162689 25155 tablet_replica.cc:330] T 31bdb5076e0e40c79ed8564c0e4bb215 P f52b8e69f724413281eff5ef630ed0fa: stopping tablet replica
I20260620 18:35:22.163096 25155 raft_consensus.cc:2229] T 31bdb5076e0e40c79ed8564c0e4bb215 P f52b8e69f724413281eff5ef630ed0fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:35:22.163638 25155 raft_consensus.cc:2258] T 31bdb5076e0e40c79ed8564c0e4bb215 P f52b8e69f724413281eff5ef630ed0fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:35:22.166457 25156 tablet_replica.cc:330] T 31bdb5076e0e40c79ed8564c0e4bb215 P 4696a52c99814944b06cc2a4542d5584: stopping tablet replica
I20260620 18:35:22.166646 25158 ts_tablet_manager.cc:1905] T 31bdb5076e0e40c79ed8564c0e4bb215 P ee6278c92e2b4dcdafaa6356dab839f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:35:22.167047 25156 raft_consensus.cc:2229] T 31bdb5076e0e40c79ed8564c0e4bb215 P 4696a52c99814944b06cc2a4542d5584 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:35:22.167517 25156 raft_consensus.cc:2258] T 31bdb5076e0e40c79ed8564c0e4bb215 P 4696a52c99814944b06cc2a4542d5584 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:35:22.169485 25155 ts_tablet_manager.cc:1905] T 31bdb5076e0e40c79ed8564c0e4bb215 P f52b8e69f724413281eff5ef630ed0fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:35:22.172407 25128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:22.136872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56034 (local address 127.10.131.1:38513)
0620 18:35:22.137065 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:22.137073 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:22.149620 (+ 12547us) server_negotiation.cc:407] Connection header received
0620 18:35:22.150193 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:22.150201 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:22.150305 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:22.150389 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:22.152764 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:22.154171 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:22.156727 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:22.157637 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:22.157757 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:22.164434 (+ 6677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:22.164448 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:22.164466 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:22.164513 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:22.168602 (+ 4089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:22.169572 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:22.169581 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:22.169585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:22.169781 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:22.170263 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:22.170268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:22.170271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:22.170794 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:22.171102 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:22.171747 (+ 645us) server_negotiation.cc:299] Negotiation successful
0620 18:35:22.171990 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:35:22.173871 24841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:22.175055 25158 ts_tablet_manager.cc:1918] T 31bdb5076e0e40c79ed8564c0e4bb215 P ee6278c92e2b4dcdafaa6356dab839f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:35:22.175308 25158 log.cc:1196] T 31bdb5076e0e40c79ed8564c0e4bb215 P ee6278c92e2b4dcdafaa6356dab839f7: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/wal/wals/31bdb5076e0e40c79ed8564c0e4bb215
I20260620 18:35:22.175977 25158 ts_tablet_manager.cc:1939] T 31bdb5076e0e40c79ed8564c0e4bb215 P ee6278c92e2b4dcdafaa6356dab839f7: Deleting consensus metadata
I20260620 18:35:22.176239 25156 ts_tablet_manager.cc:1905] T 31bdb5076e0e40c79ed8564c0e4bb215 P 4696a52c99814944b06cc2a4542d5584: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:35:22.177920 24643 catalog_manager.cc:4841] TS ee6278c92e2b4dcdafaa6356dab839f7 (127.10.131.2:42859): tablet 31bdb5076e0e40c79ed8564c0e4bb215 (table test-table [id=d7ae50109dd84b51a3848654c3124422]) successfully deleted
I20260620 18:35:22.184095 25156 ts_tablet_manager.cc:1918] T 31bdb5076e0e40c79ed8564c0e4bb215 P 4696a52c99814944b06cc2a4542d5584: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:35:22.184264 25156 log.cc:1196] T 31bdb5076e0e40c79ed8564c0e4bb215 P 4696a52c99814944b06cc2a4542d5584: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/wal/wals/31bdb5076e0e40c79ed8564c0e4bb215
I20260620 18:35:22.184819 25156 ts_tablet_manager.cc:1939] T 31bdb5076e0e40c79ed8564c0e4bb215 P 4696a52c99814944b06cc2a4542d5584: Deleting consensus metadata
I20260620 18:35:22.186733 24643 catalog_manager.cc:4841] TS 4696a52c99814944b06cc2a4542d5584 (127.10.131.3:45173): tablet 31bdb5076e0e40c79ed8564c0e4bb215 (table test-table [id=d7ae50109dd84b51a3848654c3124422]) successfully deleted
I20260620 18:35:22.283317 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 24718
I20260620 18:35:22.307478 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 24852
W20260620 18:35:22.307458 24641 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.131.1:38513 (error 108)
I20260620 18:35:22.309826 25129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:22.309436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:38513 (local address 127.0.0.1:56036)
0620 18:35:22.309592 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:22.309694 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 18:35:22.310127 24641 catalog_manager.cc:4584] TS f52b8e69f724413281eff5ef630ed0fa (127.10.131.1:38513): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 31bdb5076e0e40c79ed8564c0e4bb215: Network error: Client connection negotiation failed: client connection to 127.10.131.1:38513: connect: Connection refused (error 111)
Jun 20 18:35:22 dist-test-slave-cbn3 krb5kdc[24612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.2@KRBTEST.COM
I20260620 18:35:22.325613 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:22.308309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53706 (local address 127.10.131.2:42859)
0620 18:35:22.308499 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:22.308505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:22.308961 (+ 456us) server_negotiation.cc:407] Connection header received
0620 18:35:22.309042 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:22.309047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:22.309134 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:22.309248 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:22.311010 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:22.311814 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:22.313775 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:22.314229 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:22.314326 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:22.319446 (+ 5120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:22.319462 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:22.319477 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:22.319509 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:22.322784 (+ 3275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:22.323469 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:22.323476 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:22.323478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:22.323616 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:22.324065 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:22.324069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:22.324071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:22.324404 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:22.324753 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:22.324988 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 18:35:22.325208 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:35:22.326313 24975 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:22.363751 25129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:22.363298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:38513 (local address 127.0.0.1:56048)
0620 18:35:22.363508 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:22.363599 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 18:35:22.364754 25129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:22.364424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:38513 (local address 127.0.0.1:56052)
0620 18:35:22.364561 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:22.364647 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 18:35:22.425235 25129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:22.424737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:38513 (local address 127.0.0.1:56062)
0620 18:35:22.424981 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:22.425084 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260620 18:35:22.426230 25129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:22.425898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:38513 (local address 127.0.0.1:56072)
0620 18:35:22.426045 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:22.426123 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 18:35:22.439564 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 24852
I20260620 18:35:22.458155 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 24985
Jun 20 18:35:22 dist-test-slave-cbn3 krb5kdc[24612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.3@KRBTEST.COM
I20260620 18:35:22.473701 25126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:22.458853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55374 (local address 127.10.131.3:45173)
0620 18:35:22.459082 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:22.459086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:22.459137 (+ 51us) server_negotiation.cc:407] Connection header received
0620 18:35:22.459309 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:22.459313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:22.459378 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:22.459475 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:22.460837 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:22.461613 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:22.462795 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:22.463161 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:22.463247 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:22.467836 (+ 4589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:22.467849 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:22.467865 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:22.467899 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:22.470758 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:22.471490 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:22.471497 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:22.471499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:22.471632 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:22.471991 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:22.471998 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:22.472001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:22.472376 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:22.472686 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:22.473149 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 18:35:22.473377 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:35:22.474341 25108 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:22.506881 25129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:22.506386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:38513 (local address 127.0.0.1:56088)
0620 18:35:22.506608 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:22.506693 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 18:35:22.507903 25129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:22.507514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:38513 (local address 127.0.0.1:56092)
0620 18:35:22.507721 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:22.507799 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 18:35:22.562150 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 24985
I20260620 18:35:22.578840 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 24625
I20260620 18:35:22.580035 24687 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:22.687808 25129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:22.687064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:38513 (local address 127.0.0.1:56104)
0620 18:35:22.687274 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:22.687453 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 18:35:22.688020 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 24625
I20260620 18:35:22.709949 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:35471
--webserver_interface=127.10.131.62
--webserver_port=39297
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:35471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:23.034868 25164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:23.035135 25164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:23.035229 25164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:23.042260 25164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:35:23.042358 25164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:23.042411 25164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:23.042469 25164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:35:23.042510 25164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:35:23.052026 25164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:35471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:35471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=39297
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:35:23.053299 25164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:23.054592 25164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:23.076992 25170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:23.076925 25171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:23.076947 25173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:23.077718 25164 server_base.cc:1034] running on GCE node
I20260620 18:35:23.078666 25164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:23.080209 25164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:23.081421 25164 hybrid_clock.cc:648] HybridClock initialized: now 1781980523081380 us; error 28 us; skew 500 ppm
I20260620 18:35:23.081779 25164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:23 dist-test-slave-cbn3 krb5kdc[24612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980523, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:23.085265 25164 init.cc:377] Logged in from keytab as kudu/127.10.131.62@KRBTEST.COM (short username kudu)
I20260620 18:35:23.086925 25164 webserver.cc:460] Webserver started at http://127.10.131.62:39297/ using document root <none> and password file <none>
I20260620 18:35:23.087697 25164 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:23.087858 25164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:23.093889 25164 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 18:35:23.096719 25180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:23.098071 25164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:35:23.098251 25164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "4c29fa9c04464cf3930b9025bcf3fa5b"
format_stamp: "Formatted at 2026-06-20 18:35:20 on dist-test-slave-cbn3"
server_key: "f55fbaf3b5453f685bbb8f648227503b"
server_key_iv: "f5b7a4c0dad3df7f34ed65b7fd071f21"
server_key_version: "encryptionkey@0"
I20260620 18:35:23.099299 25164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:23.114813 25164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:23.118835 25164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:23.119149 25164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:23.138177 25164 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:35471
I20260620 18:35:23.138203 25231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:35471 every 8 connection(s)
I20260620 18:35:23.139698 25164 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:35:23.141438 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 25164
I20260620 18:35:23.142449 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.1:38513
--local_ip_for_outbound_sockets=127.10.131.1
--tserver_master_addrs=127.10.131.62:35471
--webserver_port=38873
--webserver_interface=127.10.131.1
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:35:23.149729 25232 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:35:23.153412 25232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b: Bootstrap starting.
I20260620 18:35:23.173393 25232 log.cc:824] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:23.185190 25232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:35:23.185724 25232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b: Bootstrap complete.
I20260620 18:35:23.195077 25232 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c29fa9c04464cf3930b9025bcf3fa5b" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35471 } }
I20260620 18:35:23.195936 25232 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c29fa9c04464cf3930b9025bcf3fa5b, State: Initialized, Role: FOLLOWER
I20260620 18:35:23.196560 25232 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c29fa9c04464cf3930b9025bcf3fa5b" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35471 } }
I20260620 18:35:23.196833 25232 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:35:23.196985 25232 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:35:23.197165 25232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:35:23.200965 25232 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c29fa9c04464cf3930b9025bcf3fa5b" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35471 } }
I20260620 18:35:23.201460 25232 leader_election.cc:302] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c29fa9c04464cf3930b9025bcf3fa5b; no voters:
I20260620 18:35:23.202097 25232 leader_election.cc:288] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 18:35:23.202234 25236 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [term 2 FOLLOWER]: Leader election won for term 2
I20260620 18:35:23.202975 25236 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [term 2 LEADER]: Becoming Leader. State: Replica: 4c29fa9c04464cf3930b9025bcf3fa5b, State: Running, Role: LEADER
I20260620 18:35:23.203635 25236 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c29fa9c04464cf3930b9025bcf3fa5b" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35471 } }
I20260620 18:35:23.205541 25236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c29fa9c04464cf3930b9025bcf3fa5b. Latest consensus state: current_term: 2 leader_uuid: "4c29fa9c04464cf3930b9025bcf3fa5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c29fa9c04464cf3930b9025bcf3fa5b" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35471 } } }
I20260620 18:35:23.205886 25236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:23.206367 25238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4c29fa9c04464cf3930b9025bcf3fa5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c29fa9c04464cf3930b9025bcf3fa5b" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35471 } } }
I20260620 18:35:23.206730 25238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:23.210567 25232 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:35:23.213275 25240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:35:23.221630 25240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:35:23.223026 25240 catalog_manager.cc:1259] Loaded cluster ID: 956ff27937fd49cfb92097080434316a
I20260620 18:35:23.223122 25240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:35:23.226466 25240 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 18:35:23.244714 25240 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 18:35:23.266633 25240 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:35:23.269023 25240 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b: Loaded TSK: 0
I20260620 18:35:23.271116 25240 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 18:35:23.517096 25234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:23.517406 25234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:23.517555 25234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:23.524708 25234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:23.524895 25234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:23.525079 25234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.1
I20260620 18:35:23.534690 25234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.1:38513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.131.1
--webserver_port=38873
--tserver_master_addrs=127.10.131.62:35471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.1
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:35:23.535987 25234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:23.537424 25234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:23.552160 25259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:23.552112 25260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:23.552769 25262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:23.553320 25234 server_base.cc:1034] running on GCE node
I20260620 18:35:23.554030 25234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:23.555119 25234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:23.556305 25234 hybrid_clock.cc:648] HybridClock initialized: now 1781980523556246 us; error 48 us; skew 500 ppm
I20260620 18:35:23.556680 25234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:23 dist-test-slave-cbn3 krb5kdc[24612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980523, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:23.562965 25234 init.cc:377] Logged in from keytab as kudu/127.10.131.1@KRBTEST.COM (short username kudu)
I20260620 18:35:23.564576 25234 webserver.cc:460] Webserver started at http://127.10.131.1:38873/ using document root <none> and password file <none>
I20260620 18:35:23.565317 25234 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:23.565467 25234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:23.570974 25234 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 18:35:23.574002 25269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:23.575428 25234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:35:23.575588 25234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/wal
uuid: "f52b8e69f724413281eff5ef630ed0fa"
format_stamp: "Formatted at 2026-06-20 18:35:20 on dist-test-slave-cbn3"
server_key: "4a5d3f454279ccd50a0dca085831dc67"
server_key_iv: "c9432bfe3a5f3766c2f396fa3bc0b5e2"
server_key_version: "encryptionkey@0"
I20260620 18:35:23.576515 25234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:23.599092 25234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:23.603452 25234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:23.603785 25234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:23.604905 25234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:23.606853 25276 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260620 18:35:23.612937 25276 ts_tablet_manager.cc:1852] T 31bdb5076e0e40c79ed8564c0e4bb215 P f52b8e69f724413281eff5ef630ed0fa: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260620 18:35:23.614333 25276 ts_tablet_manager.cc:1905] T 31bdb5076e0e40c79ed8564c0e4bb215 P f52b8e69f724413281eff5ef630ed0fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:35:23.622260 25276 ts_tablet_manager.cc:1918] T 31bdb5076e0e40c79ed8564c0e4bb215 P f52b8e69f724413281eff5ef630ed0fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:35:23.622471 25276 log.cc:1196] T 31bdb5076e0e40c79ed8564c0e4bb215 P f52b8e69f724413281eff5ef630ed0fa: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/wal/wals/31bdb5076e0e40c79ed8564c0e4bb215
I20260620 18:35:23.622942 25276 ts_tablet_manager.cc:1939] T 31bdb5076e0e40c79ed8564c0e4bb215 P f52b8e69f724413281eff5ef630ed0fa: Deleting consensus metadata
I20260620 18:35:23.623371 25234 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260620 18:35:23.623464 25234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.017s user 0.000s sys 0.001s
I20260620 18:35:23.623579 25234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:23.623634 25234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:23.656267 25234 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.1:38513
I20260620 18:35:23.656320 25382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.1:38513 every 8 connection(s)
I20260620 18:35:23.657588 25234 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
I20260620 18:35:23.658416 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 25234
I20260620 18:35:23.659189 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.2:42859
--local_ip_for_outbound_sockets=127.10.131.2
--tserver_master_addrs=127.10.131.62:35471
--webserver_port=41945
--webserver_interface=127.10.131.2
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:35:23 dist-test-slave-cbn3 krb5kdc[24612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980523, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:35:23.685948 25386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:23.661068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.1:47681)
0620 18:35:23.662215 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:23.662263 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:23.665114 (+ 2851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:23.666434 (+ 1320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:23.666447 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:23.667026 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:23.667547 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:23.667564 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:23.670057 (+ 2493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:23.670060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:23.671389 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:23.671404 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:23.672936 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:23.672942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:23.673129 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:23.673861 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:23.673887 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:23.676087 (+ 2200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:23.680997 (+ 4910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:23.681004 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:23.681017 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:23.681544 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:23.682095 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:23.682098 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:23.682101 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:23.682322 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:23.683313 (+ 991us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:23.683319 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:23.683664 (+ 345us) client_negotiation.cc:769] Sending connection context
0620 18:35:23.683936 (+ 272us) client_negotiation.cc:240] Negotiation successful
0620 18:35:23.684327 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"thread_start_us":617,"threads_started":1}
I20260620 18:35:23.686049 25385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:23.660959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:47681 (local address 127.10.131.62:35471)
0620 18:35:23.661627 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:23.661643 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:23.662439 (+ 796us) server_negotiation.cc:407] Connection header received
0620 18:35:23.665621 (+ 3182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:23.665634 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:23.665946 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:23.666221 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:23.667762 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:23.669888 (+ 2126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:23.671551 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:23.672636 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:23.672831 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:23.676364 (+ 3533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:23.676376 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:23.676398 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:23.676448 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:23.680829 (+ 4381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:23.681691 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:23.681702 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:23.681712 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:23.681944 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:23.682453 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:23.682459 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:23.682463 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:23.683157 (+ 694us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:23.683398 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:23.684203 (+ 805us) server_negotiation.cc:299] Negotiation successful
0620 18:35:23.684720 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":409,"threads_started":1}
I20260620 18:35:23.689270 25383 heartbeater.cc:347] Connected to a master server at 127.10.131.62:35471
I20260620 18:35:23.689625 25383 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:23.690390 25383 heartbeater.cc:511] Master 127.10.131.62:35471 requested a full tablet report, sending...
I20260620 18:35:23.692685 25197 ts_manager.cc:194] Registered new tserver with Master: f52b8e69f724413281eff5ef630ed0fa (127.10.131.1:38513)
I20260620 18:35:23.694306 25197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.1@KRBTEST.COM'} at 127.10.131.1:47681
W20260620 18:35:23.981161 25387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:23.981484 25387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:23.981666 25387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:23.988819 25387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:23.988979 25387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:23.989166 25387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.2
I20260620 18:35:23.998750 25387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.2:42859
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.131.2
--webserver_port=41945
--tserver_master_addrs=127.10.131.62:35471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.2
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:35:24.000052 25387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:24.001410 25387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:24.015337 25393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:24.015569 25392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:24.015516 25395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:24.016649 25387 server_base.cc:1034] running on GCE node
I20260620 18:35:24.017279 25387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:24.018271 25387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:24.019516 25387 hybrid_clock.cc:648] HybridClock initialized: now 1781980524019475 us; error 61 us; skew 500 ppm
I20260620 18:35:24.019833 25387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:24 dist-test-slave-cbn3 krb5kdc[24612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980524, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:24.023010 25387 init.cc:377] Logged in from keytab as kudu/127.10.131.2@KRBTEST.COM (short username kudu)
I20260620 18:35:24.024379 25387 webserver.cc:460] Webserver started at http://127.10.131.2:41945/ using document root <none> and password file <none>
I20260620 18:35:24.025060 25387 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:24.025177 25387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:24.030282 25387 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 18:35:24.032799 25402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:24.034183 25387 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:35:24.034307 25387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/wal
uuid: "ee6278c92e2b4dcdafaa6356dab839f7"
format_stamp: "Formatted at 2026-06-20 18:35:21 on dist-test-slave-cbn3"
server_key: "ed66ecdb9bad87d9d621d1da087b1b8f"
server_key_iv: "a702c8bf2ed5b0ee0bda99be877c0ffc"
server_key_version: "encryptionkey@0"
I20260620 18:35:24.035225 25387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:24.052431 25387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:24.056535 25387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:24.056758 25387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:24.057751 25387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:24.059027 25387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:24.059114 25387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:24.059211 25387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:24.059271 25387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:24.093639 25387 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.2:42859
I20260620 18:35:24.093729 25514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.2:42859 every 8 connection(s)
I20260620 18:35:24.095199 25387 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
I20260620 18:35:24.104426 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 25387
I20260620 18:35:24.105175 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.3:45173
--local_ip_for_outbound_sockets=127.10.131.3
--tserver_master_addrs=127.10.131.62:35471
--webserver_port=32829
--webserver_interface=127.10.131.3
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:35:24 dist-test-slave-cbn3 krb5kdc[24612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980524, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:35:24.115854 25385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:24.097736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:50665 (local address 127.10.131.62:35471)
0620 18:35:24.097962 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:24.097969 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:24.099084 (+ 1115us) server_negotiation.cc:407] Connection header received
0620 18:35:24.100780 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:24.100785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:24.100913 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:24.101001 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:24.102457 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:24.103191 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:24.104897 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:24.105658 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:24.105806 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:24.109407 (+ 3601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:24.109416 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:24.109424 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:24.109468 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:24.112594 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:24.113533 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:24.113537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:24.113539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:24.113657 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:24.114123 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:24.114126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:24.114128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:24.114412 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:24.114522 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:24.115395 (+ 873us) server_negotiation.cc:299] Negotiation successful
0620 18:35:24.115565 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 18:35:24.117363 25517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:24.098225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.2:50665)
0620 18:35:24.098922 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:24.098964 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:24.100537 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:24.101242 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:24.101250 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:24.101764 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:24.102306 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:24.102326 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:24.103337 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:24.103341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:24.104725 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:24.104737 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:24.105869 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:24.105874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:24.106066 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:24.106740 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:24.106763 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:24.109181 (+ 2418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:24.112733 (+ 3552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:24.112745 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:24.112763 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:24.113425 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:24.113765 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:24.113772 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:24.113778 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:24.114020 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:24.114534 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:24.114544 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:24.115004 (+ 460us) client_negotiation.cc:769] Sending connection context
0620 18:35:24.115284 (+ 280us) client_negotiation.cc:240] Negotiation successful
0620 18:35:24.115850 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":276,"threads_started":1}
I20260620 18:35:24.118785 25515 heartbeater.cc:347] Connected to a master server at 127.10.131.62:35471
I20260620 18:35:24.119091 25515 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:24.119808 25515 heartbeater.cc:511] Master 127.10.131.62:35471 requested a full tablet report, sending...
I20260620 18:35:24.121245 25197 ts_manager.cc:194] Registered new tserver with Master: ee6278c92e2b4dcdafaa6356dab839f7 (127.10.131.2:42859)
I20260620 18:35:24.122037 25197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.2@KRBTEST.COM'} at 127.10.131.2:50665
W20260620 18:35:24.435341 25518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:24.435618 25518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:24.435745 25518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:24.443301 25518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:24.443445 25518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:24.443661 25518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.3
I20260620 18:35:24.453387 25518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.3:45173
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.131.3
--webserver_port=32829
--tserver_master_addrs=127.10.131.62:35471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.3
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:35:24.454756 25518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:24.456086 25518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:24.469447 25523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:24.469379 25526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:24.469420 25524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:24.470418 25518 server_base.cc:1034] running on GCE node
I20260620 18:35:24.471009 25518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:24.471899 25518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:24.473132 25518 hybrid_clock.cc:648] HybridClock initialized: now 1781980524473097 us; error 61 us; skew 500 ppm
I20260620 18:35:24.473438 25518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:24 dist-test-slave-cbn3 krb5kdc[24612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980524, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:24.476388 25518 init.cc:377] Logged in from keytab as kudu/127.10.131.3@KRBTEST.COM (short username kudu)
I20260620 18:35:24.477800 25518 webserver.cc:460] Webserver started at http://127.10.131.3:32829/ using document root <none> and password file <none>
I20260620 18:35:24.478448 25518 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:24.478598 25518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:24.484076 25518 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 18:35:24.486845 25533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:24.488274 25518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:35:24.488407 25518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/wal
uuid: "4696a52c99814944b06cc2a4542d5584"
format_stamp: "Formatted at 2026-06-20 18:35:21 on dist-test-slave-cbn3"
server_key: "4d8aa051697d1eed27322cdb0dd34ffa"
server_key_iv: "f78c6758c8c91cbac28595683e62aaeb"
server_key_version: "encryptionkey@0"
I20260620 18:35:24.489501 25518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:24.515479 25518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:24.519589 25518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:24.519804 25518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:24.520933 25518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:24.522267 25518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:24.522357 25518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:24.522501 25518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:24.522593 25518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:24.555835 25518 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.3:45173
I20260620 18:35:24.555924 25645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.3:45173 every 8 connection(s)
I20260620 18:35:24.557260 25518 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
I20260620 18:35:24.560130 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 25518
Jun 20 18:35:24 dist-test-slave-cbn3 krb5kdc[24612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980524, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:35:24.576529 25650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:24.561328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42540 (local address 127.10.131.62:35471)
0620 18:35:24.561743 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:24.561749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:24.561766 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:35:24.561827 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:24.561832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:24.561911 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:24.562023 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:24.563450 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:24.564579 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:24.566257 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:24.566844 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:24.566971 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:24.569939 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:24.569952 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:24.569968 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:24.569996 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:24.573245 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:24.574218 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:24.574226 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:24.574229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:24.574381 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:24.574836 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:24.574843 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:24.574847 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:24.575255 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:24.575544 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:24.575771 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 18:35:24.576035 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":303,"spinlock_wait_cycles":59776,"thread_start_us":241,"threads_started":1}
I20260620 18:35:24.579777 25385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:24.559645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:49133 (local address 127.10.131.62:35471)
0620 18:35:24.559866 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:24.559873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:24.561201 (+ 1328us) server_negotiation.cc:407] Connection header received
0620 18:35:24.563337 (+ 2136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:24.563341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:24.563422 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:24.563497 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:24.565408 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:24.566396 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:24.568332 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:24.569074 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:24.569205 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:24.573131 (+ 3926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:24.573138 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:24.573143 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:24.573220 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:24.576451 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:24.577309 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:24.577313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:24.577317 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:24.577451 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:24.577959 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:24.577963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:24.577965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:24.578230 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:24.578387 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:24.579172 (+ 785us) server_negotiation.cc:299] Negotiation successful
0620 18:35:24.579402 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":65,"spinlock_wait_cycles":1408}
I20260620 18:35:24.580720 25648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:24.560219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.3:49133)
0620 18:35:24.561035 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:24.561084 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:24.562985 (+ 1901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:24.563960 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:24.563969 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:24.564617 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:24.565259 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:24.565276 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:24.566538 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:24.566543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:24.568194 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:24.568206 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:24.569267 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:24.569271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:24.569446 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:24.570076 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:24.570096 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:24.572944 (+ 2848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:24.576605 (+ 3661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:24.576616 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:24.576635 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:24.577194 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:24.577566 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:24.577569 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:24.577571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:24.577815 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:24.578368 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:24.578376 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:24.578739 (+ 363us) client_negotiation.cc:769] Sending connection context
0620 18:35:24.579069 (+ 330us) client_negotiation.cc:240] Negotiation successful
0620 18:35:24.579629 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":327,"threads_started":1}
I20260620 18:35:24.582021 25646 heartbeater.cc:347] Connected to a master server at 127.10.131.62:35471
I20260620 18:35:24.582343 25646 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:24.583056 25646 heartbeater.cc:511] Master 127.10.131.62:35471 requested a full tablet report, sending...
I20260620 18:35:24.584455 25197 ts_manager.cc:194] Registered new tserver with Master: 4696a52c99814944b06cc2a4542d5584 (127.10.131.3:45173)
I20260620 18:35:24.585461 25197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.3@KRBTEST.COM'} at 127.10.131.3:49133
I20260620 18:35:24.591318 10764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:35:24.598259 25385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:24.592292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42552 (local address 127.10.131.62:35471)
0620 18:35:24.592442 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:24.592447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:24.593068 (+ 621us) server_negotiation.cc:407] Connection header received
0620 18:35:24.593254 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:24.593258 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:24.593448 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:24.593580 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:35:24.593946 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:24.594660 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:24.596275 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:24.596658 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:24.596748 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:24.597364 (+ 616us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:35:24.597818 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:35:24.597919 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:24.598011 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 18:35:24.598073 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 18:35:24.599200 25197 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:42552:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:35:24.607250 25197 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:35:24.637889 25658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:24.619756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:38513 (local address 127.0.0.1:56108)
0620 18:35:24.621419 (+ 1663us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:24.621470 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:24.621698 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:24.624095 (+ 2397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:24.624107 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:24.624161 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:24.624673 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:24.624691 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:24.628858 (+ 4167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:24.628889 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:24.633536 (+ 4647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:24.633558 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:24.636740 (+ 3182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:24.636746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:24.637011 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:24.637078 (+ 67us) client_negotiation.cc:769] Sending connection context
0620 18:35:24.637210 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 18:35:24.637456 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1342,"thread_start_us":13817,"threads_started":1}
I20260620 18:35:24.638903 25657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:24.619889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56108 (local address 127.10.131.1:38513)
0620 18:35:24.620726 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:24.620735 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:24.621904 (+ 1169us) server_negotiation.cc:407] Connection header received
0620 18:35:24.622032 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:24.622041 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:24.622304 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:24.625026 (+ 2722us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:24.625347 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:24.628636 (+ 3289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:24.633786 (+ 5150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:24.636136 (+ 2350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:24.636269 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:24.636516 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:24.638409 (+ 1893us) server_negotiation.cc:299] Negotiation successful
0620 18:35:24.638581 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":928,"threads_started":1}
I20260620 18:35:24.641103 25659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:24.620990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53712 (local address 127.10.131.2:42859)
0620 18:35:24.634565 (+ 13575us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:24.634581 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:24.634605 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:35:24.634687 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:24.634693 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:24.634884 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:24.635087 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:24.635677 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:24.637118 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:24.638836 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:24.640028 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:24.640198 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:24.640377 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:24.640637 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 18:35:24.640789 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13428,"thread_start_us":13338,"threads_started":1}
I20260620 18:35:24.641108 25660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:24.620443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:42859 (local address 127.0.0.1:53712)
0620 18:35:24.632972 (+ 12529us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:24.632997 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:24.633138 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:24.635067 (+ 1929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:24.635072 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:24.635097 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:24.635478 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:24.635489 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:24.637260 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:24.637265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:24.638689 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:24.638699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:24.640279 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:24.640285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:24.640461 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:24.640483 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:35:24.640584 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 18:35:24.640733 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12332,"thread_start_us":539,"threads_started":1}
I20260620 18:35:24.641076 25318 tablet_service.cc:1467] Processing CreateTablet for tablet b9d2b93752d9412bb67fe9b77c9144c6 (DEFAULT_TABLE table=test-table [id=f38fe047fde5484c906ac40daef636d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:24.642627 25318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d2b93752d9412bb67fe9b77c9144c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:24.644104 25656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:24.621882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:45173 (local address 127.0.0.1:55380)
0620 18:35:24.633924 (+ 12042us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:24.633945 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:24.634067 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:24.636349 (+ 2282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:24.636354 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:24.636377 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:24.636771 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:24.636783 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:24.639080 (+ 2297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:24.639086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:24.641390 (+ 2304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:24.641405 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:24.643387 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:24.643393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:24.643552 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:24.643583 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 18:35:24.643663 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 18:35:24.643794 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11870,"thread_start_us":11349,"threads_started":1}
I20260620 18:35:24.645115 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:24.622198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55380 (local address 127.10.131.3:45173)
0620 18:35:24.635796 (+ 13598us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:24.635806 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:24.635848 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:35:24.635934 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:24.635940 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:24.636154 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:24.636990 (+ 836us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:24.637285 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:24.638912 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:24.641615 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:24.643171 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:24.643307 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:24.643531 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:24.644589 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0620 18:35:24.644761 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13405,"thread_start_us":13234,"threads_started":1}
I20260620 18:35:24.644758 25450 tablet_service.cc:1467] Processing CreateTablet for tablet b9d2b93752d9412bb67fe9b77c9144c6 (DEFAULT_TABLE table=test-table [id=f38fe047fde5484c906ac40daef636d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:24.645978 25450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d2b93752d9412bb67fe9b77c9144c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:24.648748 25581 tablet_service.cc:1467] Processing CreateTablet for tablet b9d2b93752d9412bb67fe9b77c9144c6 (DEFAULT_TABLE table=test-table [id=f38fe047fde5484c906ac40daef636d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:24.650051 25581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d2b93752d9412bb67fe9b77c9144c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:24.661237 25662 tablet_bootstrap.cc:492] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa: Bootstrap starting.
I20260620 18:35:24.664999 25664 tablet_bootstrap.cc:492] T b9d2b93752d9412bb67fe9b77c9144c6 P 4696a52c99814944b06cc2a4542d5584: Bootstrap starting.
I20260620 18:35:24.665076 25662 tablet_bootstrap.cc:654] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:24.666191 25662 log.cc:824] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:24.668658 25664 tablet_bootstrap.cc:654] T b9d2b93752d9412bb67fe9b77c9144c6 P 4696a52c99814944b06cc2a4542d5584: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:24.669974 25664 log.cc:824] T b9d2b93752d9412bb67fe9b77c9144c6 P 4696a52c99814944b06cc2a4542d5584: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:24.671965 25662 tablet_bootstrap.cc:492] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa: No bootstrap required, opened a new log
I20260620 18:35:24.672377 25662 ts_tablet_manager.cc:1397] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260620 18:35:24.678906 25663 tablet_bootstrap.cc:492] T b9d2b93752d9412bb67fe9b77c9144c6 P ee6278c92e2b4dcdafaa6356dab839f7: Bootstrap starting.
I20260620 18:35:24.683377 25663 tablet_bootstrap.cc:654] T b9d2b93752d9412bb67fe9b77c9144c6 P ee6278c92e2b4dcdafaa6356dab839f7: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:24.684017 25662 raft_consensus.cc:348] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f52b8e69f724413281eff5ef630ed0fa" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38513 } } peers { permanent_uuid: "ee6278c92e2b4dcdafaa6356dab839f7" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 42859 } } peers { permanent_uuid: "4696a52c99814944b06cc2a4542d5584" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 45173 } }
I20260620 18:35:24.684490 25664 tablet_bootstrap.cc:492] T b9d2b93752d9412bb67fe9b77c9144c6 P 4696a52c99814944b06cc2a4542d5584: No bootstrap required, opened a new log
I20260620 18:35:24.684507 25662 raft_consensus.cc:374] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:24.684593 25662 raft_consensus.cc:729] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f52b8e69f724413281eff5ef630ed0fa, State: Initialized, Role: FOLLOWER
I20260620 18:35:24.684782 25664 ts_tablet_manager.cc:1397] T b9d2b93752d9412bb67fe9b77c9144c6 P 4696a52c99814944b06cc2a4542d5584: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.014s
I20260620 18:35:24.685287 25662 consensus_queue.cc:260] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f52b8e69f724413281eff5ef630ed0fa" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38513 } } peers { permanent_uuid: "ee6278c92e2b4dcdafaa6356dab839f7" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 42859 } } peers { permanent_uuid: "4696a52c99814944b06cc2a4542d5584" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 45173 } }
I20260620 18:35:24.685667 25663 log.cc:824] T b9d2b93752d9412bb67fe9b77c9144c6 P ee6278c92e2b4dcdafaa6356dab839f7: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:24.686470 25383 heartbeater.cc:503] Master 127.10.131.62:35471 was elected leader, sending a full tablet report...
I20260620 18:35:24.686920 25662 ts_tablet_manager.cc:1428] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260620 18:35:24.689709 25663 tablet_bootstrap.cc:492] T b9d2b93752d9412bb67fe9b77c9144c6 P ee6278c92e2b4dcdafaa6356dab839f7: No bootstrap required, opened a new log
I20260620 18:35:24.689977 25663 ts_tablet_manager.cc:1397] T b9d2b93752d9412bb67fe9b77c9144c6 P ee6278c92e2b4dcdafaa6356dab839f7: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260620 18:35:24.697893 25664 raft_consensus.cc:348] T b9d2b93752d9412bb67fe9b77c9144c6 P 4696a52c99814944b06cc2a4542d5584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f52b8e69f724413281eff5ef630ed0fa" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38513 } } peers { permanent_uuid: "ee6278c92e2b4dcdafaa6356dab839f7" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 42859 } } peers { permanent_uuid: "4696a52c99814944b06cc2a4542d5584" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 45173 } }
I20260620 18:35:24.698341 25664 raft_consensus.cc:374] T b9d2b93752d9412bb67fe9b77c9144c6 P 4696a52c99814944b06cc2a4542d5584 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:24.698437 25664 raft_consensus.cc:729] T b9d2b93752d9412bb67fe9b77c9144c6 P 4696a52c99814944b06cc2a4542d5584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4696a52c99814944b06cc2a4542d5584, State: Initialized, Role: FOLLOWER
I20260620 18:35:24.699069 25664 consensus_queue.cc:260] T b9d2b93752d9412bb67fe9b77c9144c6 P 4696a52c99814944b06cc2a4542d5584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f52b8e69f724413281eff5ef630ed0fa" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38513 } } peers { permanent_uuid: "ee6278c92e2b4dcdafaa6356dab839f7" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 42859 } } peers { permanent_uuid: "4696a52c99814944b06cc2a4542d5584" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 45173 } }
I20260620 18:35:24.700366 25646 heartbeater.cc:503] Master 127.10.131.62:35471 was elected leader, sending a full tablet report...
I20260620 18:35:24.701097 25663 raft_consensus.cc:348] T b9d2b93752d9412bb67fe9b77c9144c6 P ee6278c92e2b4dcdafaa6356dab839f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f52b8e69f724413281eff5ef630ed0fa" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38513 } } peers { permanent_uuid: "ee6278c92e2b4dcdafaa6356dab839f7" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 42859 } } peers { permanent_uuid: "4696a52c99814944b06cc2a4542d5584" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 45173 } }
I20260620 18:35:24.701534 25663 raft_consensus.cc:374] T b9d2b93752d9412bb67fe9b77c9144c6 P ee6278c92e2b4dcdafaa6356dab839f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:24.701632 25663 raft_consensus.cc:729] T b9d2b93752d9412bb67fe9b77c9144c6 P ee6278c92e2b4dcdafaa6356dab839f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee6278c92e2b4dcdafaa6356dab839f7, State: Initialized, Role: FOLLOWER
I20260620 18:35:24.702239 25664 ts_tablet_manager.cc:1428] T b9d2b93752d9412bb67fe9b77c9144c6 P 4696a52c99814944b06cc2a4542d5584: Time spent starting tablet: real 0.017s user 0.009s sys 0.005s
I20260620 18:35:24.702276 25663 consensus_queue.cc:260] T b9d2b93752d9412bb67fe9b77c9144c6 P ee6278c92e2b4dcdafaa6356dab839f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f52b8e69f724413281eff5ef630ed0fa" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38513 } } peers { permanent_uuid: "ee6278c92e2b4dcdafaa6356dab839f7" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 42859 } } peers { permanent_uuid: "4696a52c99814944b06cc2a4542d5584" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 45173 } }
I20260620 18:35:24.703506 25515 heartbeater.cc:503] Master 127.10.131.62:35471 was elected leader, sending a full tablet report...
I20260620 18:35:24.704185 25663 ts_tablet_manager.cc:1428] T b9d2b93752d9412bb67fe9b77c9144c6 P ee6278c92e2b4dcdafaa6356dab839f7: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260620 18:35:24.770409 25669 raft_consensus.cc:482] T b9d2b93752d9412bb67fe9b77c9144c6 P 4696a52c99814944b06cc2a4542d5584 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:35:24.770679 25669 raft_consensus.cc:504] T b9d2b93752d9412bb67fe9b77c9144c6 P 4696a52c99814944b06cc2a4542d5584 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f52b8e69f724413281eff5ef630ed0fa" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38513 } } peers { permanent_uuid: "ee6278c92e2b4dcdafaa6356dab839f7" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 42859 } } peers { permanent_uuid: "4696a52c99814944b06cc2a4542d5584" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 45173 } }
I20260620 18:35:24.772212 25669 leader_election.cc:288] T b9d2b93752d9412bb67fe9b77c9144c6 P 4696a52c99814944b06cc2a4542d5584 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f52b8e69f724413281eff5ef630ed0fa (127.10.131.1:38513), ee6278c92e2b4dcdafaa6356dab839f7 (127.10.131.2:42859)
I20260620 18:35:24.778896 25659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:24.772779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:59193 (local address 127.10.131.2:42859)
0620 18:35:24.773007 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:24.773013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:24.773160 (+ 147us) server_negotiation.cc:407] Connection header received
0620 18:35:24.773356 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:24.773362 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:24.773450 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:24.773543 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:24.774201 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:24.774994 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:24.776828 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:24.778037 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:24.778153 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:24.778203 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:24.778610 (+ 407us) server_negotiation.cc:299] Negotiation successful
0620 18:35:24.778713 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 18:35:24.778986 25648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:24.772744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:42859 (local address 127.10.131.3:59193)
0620 18:35:24.772989 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:24.773010 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:24.773169 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:24.773746 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:24.773754 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:24.773795 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:24.774039 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:24.774051 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:24.775115 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:24.775121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:24.776675 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:24.776686 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:24.778202 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:24.778208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:24.778414 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:24.778450 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 18:35:24.778562 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 18:35:24.778700 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 18:35:24.780067 25657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:24.772700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:38763 (local address 127.10.131.1:38513)
0620 18:35:24.773091 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:24.773098 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:24.773940 (+ 842us) server_negotiation.cc:407] Connection header received
0620 18:35:24.774102 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:24.774110 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:24.774198 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:24.774279 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:24.774876 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:24.775865 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:24.777946 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:24.779222 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:24.779363 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:24.779427 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:24.779725 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 18:35:24.779858 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214}
I20260620 18:35:24.780153 25671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:24.772783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:38513 (local address 127.10.131.3:38763)
0620 18:35:24.773806 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:24.773834 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:24.773962 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:24.774376 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:24.774381 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:24.774407 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:24.774716 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:24.774724 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:24.776008 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:24.776012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:24.777758 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:24.777773 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:24.779374 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:24.779379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:24.779534 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:24.779585 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 18:35:24.779674 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 18:35:24.779794 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":879,"mutex_wait_us":1,"thread_start_us":632,"threads_started":1}
I20260620 18:35:24.780100 25470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d2b93752d9412bb67fe9b77c9144c6" candidate_uuid: "4696a52c99814944b06cc2a4542d5584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee6278c92e2b4dcdafaa6356dab839f7" is_pre_election: true
I20260620 18:35:24.780700 25470 raft_consensus.cc:2454] T b9d2b93752d9412bb67fe9b77c9144c6 P ee6278c92e2b4dcdafaa6356dab839f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4696a52c99814944b06cc2a4542d5584 in term 0.
I20260620 18:35:24.781214 25338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d2b93752d9412bb67fe9b77c9144c6" candidate_uuid: "4696a52c99814944b06cc2a4542d5584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f52b8e69f724413281eff5ef630ed0fa" is_pre_election: true
I20260620 18:35:24.781497 25536 leader_election.cc:302] T b9d2b93752d9412bb67fe9b77c9144c6 P 4696a52c99814944b06cc2a4542d5584 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4696a52c99814944b06cc2a4542d5584, ee6278c92e2b4dcdafaa6356dab839f7; no voters:
I20260620 18:35:24.781742 25338 raft_consensus.cc:2454] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4696a52c99814944b06cc2a4542d5584 in term 0.
I20260620 18:35:24.781867 25669 raft_consensus.cc:2781] T b9d2b93752d9412bb67fe9b77c9144c6 P 4696a52c99814944b06cc2a4542d5584 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:35:24.782061 25669 raft_consensus.cc:482] T b9d2b93752d9412bb67fe9b77c9144c6 P 4696a52c99814944b06cc2a4542d5584 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:35:24.782176 25669 raft_consensus.cc:3037] T b9d2b93752d9412bb67fe9b77c9144c6 P 4696a52c99814944b06cc2a4542d5584 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:24.784341 25669 raft_consensus.cc:504] T b9d2b93752d9412bb67fe9b77c9144c6 P 4696a52c99814944b06cc2a4542d5584 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f52b8e69f724413281eff5ef630ed0fa" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38513 } } peers { permanent_uuid: "ee6278c92e2b4dcdafaa6356dab839f7" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 42859 } } peers { permanent_uuid: "4696a52c99814944b06cc2a4542d5584" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 45173 } }
I20260620 18:35:24.784961 25669 leader_election.cc:288] T b9d2b93752d9412bb67fe9b77c9144c6 P 4696a52c99814944b06cc2a4542d5584 [CANDIDATE]: Term 1 election: Requested vote from peers f52b8e69f724413281eff5ef630ed0fa (127.10.131.1:38513), ee6278c92e2b4dcdafaa6356dab839f7 (127.10.131.2:42859)
I20260620 18:35:24.785390 25470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d2b93752d9412bb67fe9b77c9144c6" candidate_uuid: "4696a52c99814944b06cc2a4542d5584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee6278c92e2b4dcdafaa6356dab839f7"
I20260620 18:35:24.785390 25338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d2b93752d9412bb67fe9b77c9144c6" candidate_uuid: "4696a52c99814944b06cc2a4542d5584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f52b8e69f724413281eff5ef630ed0fa"
I20260620 18:35:24.785609 25338 raft_consensus.cc:3037] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:24.785609 25470 raft_consensus.cc:3037] T b9d2b93752d9412bb67fe9b77c9144c6 P ee6278c92e2b4dcdafaa6356dab839f7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:24.788774 25338 raft_consensus.cc:2454] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4696a52c99814944b06cc2a4542d5584 in term 1.
I20260620 18:35:24.788774 25470 raft_consensus.cc:2454] T b9d2b93752d9412bb67fe9b77c9144c6 P ee6278c92e2b4dcdafaa6356dab839f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4696a52c99814944b06cc2a4542d5584 in term 1.
I20260620 18:35:24.789167 25534 leader_election.cc:302] T b9d2b93752d9412bb67fe9b77c9144c6 P 4696a52c99814944b06cc2a4542d5584 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4696a52c99814944b06cc2a4542d5584, f52b8e69f724413281eff5ef630ed0fa; no voters:
I20260620 18:35:24.789460 25669 raft_consensus.cc:2781] T b9d2b93752d9412bb67fe9b77c9144c6 P 4696a52c99814944b06cc2a4542d5584 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:35:24.789875 25669 raft_consensus.cc:686] T b9d2b93752d9412bb67fe9b77c9144c6 P 4696a52c99814944b06cc2a4542d5584 [term 1 LEADER]: Becoming Leader. State: Replica: 4696a52c99814944b06cc2a4542d5584, State: Running, Role: LEADER
I20260620 18:35:24.790306 25669 consensus_queue.cc:237] T b9d2b93752d9412bb67fe9b77c9144c6 P 4696a52c99814944b06cc2a4542d5584 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f52b8e69f724413281eff5ef630ed0fa" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38513 } } peers { permanent_uuid: "ee6278c92e2b4dcdafaa6356dab839f7" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 42859 } } peers { permanent_uuid: "4696a52c99814944b06cc2a4542d5584" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 45173 } }
I20260620 18:35:24.795161 25196 catalog_manager.cc:5495] T b9d2b93752d9412bb67fe9b77c9144c6 P 4696a52c99814944b06cc2a4542d5584 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4696a52c99814944b06cc2a4542d5584 (127.10.131.3). New cstate: current_term: 1 leader_uuid: "4696a52c99814944b06cc2a4542d5584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f52b8e69f724413281eff5ef630ed0fa" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee6278c92e2b4dcdafaa6356dab839f7" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 42859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4696a52c99814944b06cc2a4542d5584" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 45173 } health_report { overall_health: HEALTHY } } }
W20260620 18:35:24.809093 25647 tablet.cc:2369] T b9d2b93752d9412bb67fe9b77c9144c6 P 4696a52c99814944b06cc2a4542d5584: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:35:24.823598 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:24.818382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55392 (local address 127.10.131.3:45173)
0620 18:35:24.818591 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:24.818595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:24.818613 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:35:24.818757 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:24.818761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:24.818839 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:24.818959 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:35:24.819305 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:24.820015 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:24.821469 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:24.821853 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:24.821978 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:24.822506 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:35:24.823071 (+ 565us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:35:24.823179 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:24.823327 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 18:35:24.823398 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 18:35:24.831930 25338 raft_consensus.cc:1261] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [term 1 FOLLOWER]: Refusing update from remote peer 4696a52c99814944b06cc2a4542d5584: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:35:24.831931 25470 raft_consensus.cc:1261] T b9d2b93752d9412bb67fe9b77c9144c6 P ee6278c92e2b4dcdafaa6356dab839f7 [term 1 FOLLOWER]: Refusing update from remote peer 4696a52c99814944b06cc2a4542d5584: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:35:24.832975 25672 consensus_queue.cc:1035] T b9d2b93752d9412bb67fe9b77c9144c6 P 4696a52c99814944b06cc2a4542d5584 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee6278c92e2b4dcdafaa6356dab839f7" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 42859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:35:24.833339 25669 consensus_queue.cc:1035] T b9d2b93752d9412bb67fe9b77c9144c6 P 4696a52c99814944b06cc2a4542d5584 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f52b8e69f724413281eff5ef630ed0fa" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:35:24.844372 25675 mvcc.cc:205] Tried to move back new op lower bound from 7298992229702152192 to 7298992229545496576. Current Snapshot: MvccSnapshot[applied={T|T < 7298992229702152192}]
I20260620 18:35:24.845185 25676 mvcc.cc:205] Tried to move back new op lower bound from 7298992229702152192 to 7298992229545496576. Current Snapshot: MvccSnapshot[applied={T|T < 7298992229702152192}]
I20260620 18:35:24.859732 25196 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:42552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:35:24.860023 25196 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:42552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:35:24.863268 25196 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b: Sending DeleteTablet for 3 replicas of tablet b9d2b93752d9412bb67fe9b77c9144c6
I20260620 18:35:24.864509 25318 tablet_service.cc:1514] Processing DeleteTablet for tablet b9d2b93752d9412bb67fe9b77c9144c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:35:24 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:56108
I20260620 18:35:24.864529 25450 tablet_service.cc:1514] Processing DeleteTablet for tablet b9d2b93752d9412bb67fe9b77c9144c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:35:24 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:53712
I20260620 18:35:24.865511 25581 tablet_service.cc:1514] Processing DeleteTablet for tablet b9d2b93752d9412bb67fe9b77c9144c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:35:24 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:55380
I20260620 18:35:24.869297 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 25234
I20260620 18:35:24.875449 25687 tablet_replica.cc:330] T b9d2b93752d9412bb67fe9b77c9144c6 P 4696a52c99814944b06cc2a4542d5584: stopping tablet replica
I20260620 18:35:24.875988 25687 raft_consensus.cc:2229] T b9d2b93752d9412bb67fe9b77c9144c6 P 4696a52c99814944b06cc2a4542d5584 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:35:24.876813 25687 raft_consensus.cc:2258] T b9d2b93752d9412bb67fe9b77c9144c6 P 4696a52c99814944b06cc2a4542d5584 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:35:24.879179 25686 tablet_replica.cc:330] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa: stopping tablet replica
I20260620 18:35:24.880139 25686 raft_consensus.cc:2229] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:35:24.882339 25687 ts_tablet_manager.cc:1905] T b9d2b93752d9412bb67fe9b77c9144c6 P 4696a52c99814944b06cc2a4542d5584: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:35:24.885139 25686 raft_consensus.cc:2258] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:35:24.886224 25685 tablet_replica.cc:330] T b9d2b93752d9412bb67fe9b77c9144c6 P ee6278c92e2b4dcdafaa6356dab839f7: stopping tablet replica
I20260620 18:35:24.887501 25685 raft_consensus.cc:2229] T b9d2b93752d9412bb67fe9b77c9144c6 P ee6278c92e2b4dcdafaa6356dab839f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:35:24.888043 25685 raft_consensus.cc:2258] T b9d2b93752d9412bb67fe9b77c9144c6 P ee6278c92e2b4dcdafaa6356dab839f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:35:24.888417 25686 ts_tablet_manager.cc:1905] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:35:24.890198 25685 ts_tablet_manager.cc:1905] T b9d2b93752d9412bb67fe9b77c9144c6 P ee6278c92e2b4dcdafaa6356dab839f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:35:24.890492 25687 ts_tablet_manager.cc:1918] T b9d2b93752d9412bb67fe9b77c9144c6 P 4696a52c99814944b06cc2a4542d5584: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:35:24.890672 25687 log.cc:1196] T b9d2b93752d9412bb67fe9b77c9144c6 P 4696a52c99814944b06cc2a4542d5584: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/wal/wals/b9d2b93752d9412bb67fe9b77c9144c6
I20260620 18:35:24.891280 25687 ts_tablet_manager.cc:1939] T b9d2b93752d9412bb67fe9b77c9144c6 P 4696a52c99814944b06cc2a4542d5584: Deleting consensus metadata
I20260620 18:35:24.893009 25657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:24.870311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56122 (local address 127.10.131.1:38513)
0620 18:35:24.871059 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:24.871067 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:24.871090 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:35:24.871160 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:24.871169 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:24.871265 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:24.871385 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:24.874263 (+ 2878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:24.875606 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:24.877892 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:24.878856 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:24.879016 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:24.882686 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:24.882700 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:24.882716 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:24.882761 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:24.886581 (+ 3820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:24.890633 (+ 4052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:24.890642 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:24.890647 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:24.890828 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:24.891270 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:24.891278 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:24.891281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:24.891769 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:24.892072 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:24.892410 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 18:35:24.892634 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578}
I20260620 18:35:24.893105 25183 catalog_manager.cc:4841] TS 4696a52c99814944b06cc2a4542d5584 (127.10.131.3:45173): tablet b9d2b93752d9412bb67fe9b77c9144c6 (table test-table [id=f38fe047fde5484c906ac40daef636d0]) successfully deleted
I20260620 18:35:24.893683 25378 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:24.899466 25685 ts_tablet_manager.cc:1918] T b9d2b93752d9412bb67fe9b77c9144c6 P ee6278c92e2b4dcdafaa6356dab839f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:35:24.899637 25685 log.cc:1196] T b9d2b93752d9412bb67fe9b77c9144c6 P ee6278c92e2b4dcdafaa6356dab839f7: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/wal/wals/b9d2b93752d9412bb67fe9b77c9144c6
I20260620 18:35:24.900207 25685 ts_tablet_manager.cc:1939] T b9d2b93752d9412bb67fe9b77c9144c6 P ee6278c92e2b4dcdafaa6356dab839f7: Deleting consensus metadata
I20260620 18:35:24.902097 25183 catalog_manager.cc:4841] TS ee6278c92e2b4dcdafaa6356dab839f7 (127.10.131.2:42859): tablet b9d2b93752d9412bb67fe9b77c9144c6 (table test-table [id=f38fe047fde5484c906ac40daef636d0]) successfully deleted
I20260620 18:35:24.983403 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 25234
I20260620 18:35:25.002007 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 25387
W20260620 18:35:25.002246 25181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.131.1:38513 (error 108)
I20260620 18:35:25.004243 25656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:25.003673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:38513 (local address 127.0.0.1:56134)
0620 18:35:25.003914 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:25.004058 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260620 18:35:25.004509 25181 catalog_manager.cc:4584] TS f52b8e69f724413281eff5ef630ed0fa (127.10.131.1:38513): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b9d2b93752d9412bb67fe9b77c9144c6: Network error: Client connection negotiation failed: client connection to 127.10.131.1:38513: connect: Connection refused (error 111)
I20260620 18:35:25.017712 25659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:25.003150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53714 (local address 127.10.131.2:42859)
0620 18:35:25.003329 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:25.003335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:25.003354 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:35:25.003426 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:25.003431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:25.003524 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:25.003647 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:25.005414 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:25.006145 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:25.007433 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:25.007840 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:25.008001 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:25.011029 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:25.011043 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:25.011057 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:25.011093 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:25.013974 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:25.014744 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:25.014759 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:25.014762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:25.014884 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:25.015363 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:25.015369 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:25.015372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:25.015906 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:25.016247 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:25.017123 (+ 876us) server_negotiation.cc:299] Negotiation successful
0620 18:35:25.017343 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:35:25.018610 25510 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:25.058218 25656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:25.057641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:38513 (local address 127.0.0.1:56148)
0620 18:35:25.057889 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:25.058018 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 18:35:25.059376 25656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:25.058977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:38513 (local address 127.0.0.1:56154)
0620 18:35:25.059138 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:25.059226 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 18:35:25.107378 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 25387
I20260620 18:35:25.120201 25656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:25.119620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:38513 (local address 127.0.0.1:56160)
0620 18:35:25.119861 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:25.119993 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 18:35:25.121491 25656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:25.121037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:38513 (local address 127.0.0.1:56174)
0620 18:35:25.121227 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:25.121324 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 18:35:25.125288 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 25518
I20260620 18:35:25.141682 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:25.126083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55398 (local address 127.10.131.3:45173)
0620 18:35:25.126275 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:25.126282 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:25.126305 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:35:25.126434 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:25.126439 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:25.126556 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:25.126680 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:25.128056 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:25.129303 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:25.130625 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:25.131342 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:25.131482 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:25.134175 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:25.134188 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:25.134214 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:25.134263 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:25.138422 (+ 4159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:25.139296 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:25.139307 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:25.139311 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:25.139496 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:25.139901 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:25.139907 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:25.139910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:25.140392 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:25.140727 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:25.141023 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 18:35:25.141247 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:35:25.142592 25641 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:25.202178 25656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:25.201664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:38513 (local address 127.0.0.1:56182)
0620 18:35:25.201924 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:25.202019 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260620 18:35:25.203230 25656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:25.202852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:38513 (local address 127.0.0.1:56192)
0620 18:35:25.203043 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:25.203119 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260620 18:35:25.264353 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 25518
I20260620 18:35:25.284461 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 25164
I20260620 18:35:25.285584 25227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:25.396948 25656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:25.396212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:38513 (local address 127.0.0.1:56208)
0620 18:35:25.396450 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:25.396578 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 18:35:25.397153 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 25164
I20260620 18:35:25.415627 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:35471
--webserver_interface=127.10.131.62
--webserver_port=39297
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:35471
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:25.737053 25693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:25.737330 25693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:25.737516 25693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:25.744653 25693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:35:25.744762 25693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:25.744885 25693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:25.744998 25693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:35:25.745082 25693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:35:25.754937 25693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:35471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:35471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=39297
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:35:25.756737 25693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:25.758090 25693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:25.772436 25701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:25.772473 25698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:25.773020 25699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:25.773382 25693 server_base.cc:1034] running on GCE node
I20260620 18:35:25.774071 25693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:25.775352 25693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:25.776546 25693 hybrid_clock.cc:648] HybridClock initialized: now 1781980525776512 us; error 26 us; skew 500 ppm
I20260620 18:35:25.776978 25693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:25 dist-test-slave-cbn3 krb5kdc[24612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980525, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:25.779878 25693 init.cc:377] Logged in from keytab as kudu/127.10.131.62@KRBTEST.COM (short username kudu)
I20260620 18:35:25.781291 25693 webserver.cc:460] Webserver started at http://127.10.131.62:39297/ using document root <none> and password file <none>
I20260620 18:35:25.781921 25693 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:25.782020 25693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:25.787341 25693 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 18:35:25.789986 25708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:25.791271 25693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:35:25.791406 25693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "4c29fa9c04464cf3930b9025bcf3fa5b"
format_stamp: "Formatted at 2026-06-20 18:35:20 on dist-test-slave-cbn3"
server_key: "f55fbaf3b5453f685bbb8f648227503b"
server_key_iv: "f5b7a4c0dad3df7f34ed65b7fd071f21"
server_key_version: "encryptionkey@0"
I20260620 18:35:25.792289 25693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:25.810434 25693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:25.814854 25693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:25.815064 25693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:25.833241 25693 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:35471
I20260620 18:35:25.833262 25759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:35471 every 8 connection(s)
I20260620 18:35:25.834721 25693 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:35:25.838323 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 25693
I20260620 18:35:25.839133 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.1:38513
--local_ip_for_outbound_sockets=127.10.131.1
--tserver_master_addrs=127.10.131.62:35471
--webserver_port=38873
--webserver_interface=127.10.131.1
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:35:25.841555 25760 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:35:25.844878 25760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b: Bootstrap starting.
I20260620 18:35:25.875706 25760 log.cc:824] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:25.895639 25760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:35:25.896281 25760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b: Bootstrap complete.
I20260620 18:35:25.905493 25760 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c29fa9c04464cf3930b9025bcf3fa5b" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35471 } }
I20260620 18:35:25.906407 25760 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c29fa9c04464cf3930b9025bcf3fa5b, State: Initialized, Role: FOLLOWER
I20260620 18:35:25.907083 25760 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c29fa9c04464cf3930b9025bcf3fa5b" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35471 } }
I20260620 18:35:25.907323 25760 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:35:25.907434 25760 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:35:25.907627 25760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [term 2 FOLLOWER]: Advancing to term 3
I20260620 18:35:25.911446 25760 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c29fa9c04464cf3930b9025bcf3fa5b" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35471 } }
I20260620 18:35:25.911940 25760 leader_election.cc:302] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c29fa9c04464cf3930b9025bcf3fa5b; no voters:
I20260620 18:35:25.912554 25760 leader_election.cc:288] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 18:35:25.912755 25764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [term 3 FOLLOWER]: Leader election won for term 3
I20260620 18:35:25.913832 25764 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [term 3 LEADER]: Becoming Leader. State: Replica: 4c29fa9c04464cf3930b9025bcf3fa5b, State: Running, Role: LEADER
I20260620 18:35:25.914394 25764 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c29fa9c04464cf3930b9025bcf3fa5b" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35471 } }
I20260620 18:35:25.914723 25760 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:35:25.916805 25765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4c29fa9c04464cf3930b9025bcf3fa5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c29fa9c04464cf3930b9025bcf3fa5b" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35471 } } }
I20260620 18:35:25.917181 25765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:25.922056 25766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c29fa9c04464cf3930b9025bcf3fa5b. Latest consensus state: current_term: 3 leader_uuid: "4c29fa9c04464cf3930b9025bcf3fa5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c29fa9c04464cf3930b9025bcf3fa5b" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 35471 } } }
I20260620 18:35:25.922466 25766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c29fa9c04464cf3930b9025bcf3fa5b [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:25.923287 25769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:35:25.933830 25769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:35:25.935804 25769 catalog_manager.cc:1259] Loaded cluster ID: 956ff27937fd49cfb92097080434316a
I20260620 18:35:25.935985 25769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 18:35:25.938700 25769 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff0b446b9ad google::LogMessage::Fail() at ??:0
@ 0x7ff0b446e88a google::LogMessage::SendToLog() at ??:0
@ 0x7ff0b446b4de google::LogMessage::Flush() at ??:0
@ 0x7ff0b446ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff0c9e9e512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7ff0c9e9b976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff0c9f23610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7ff0c9f23439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff0b70b7345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff0b70ba91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7ff0b70ba799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff0b7095eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff0ba36f6db start_thread at ??:0
@ 0x7ff0b330561f clone at ??:0
W20260620 18:35:26.235314 25762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:26.235646 25762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:26.235744 25762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:26.243079 25762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:26.243201 25762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:26.243387 25762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.1
I20260620 18:35:26.253180 25762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.1:38513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.131.1
--webserver_port=38873
--tserver_master_addrs=127.10.131.62:35471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.1
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:35:26.254586 25762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:26.255949 25762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:26.269258 25801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:26.269286 25804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:26.269227 25802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:26.270066 25762 server_base.cc:1034] running on GCE node
I20260620 18:35:26.270742 25762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:26.271816 25762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:26.273038 25762 hybrid_clock.cc:648] HybridClock initialized: now 1781980526272986 us; error 47 us; skew 500 ppm
I20260620 18:35:26.273366 25762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:26 dist-test-slave-cbn3 krb5kdc[24612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980526, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:26.276540 25762 init.cc:377] Logged in from keytab as kudu/127.10.131.1@KRBTEST.COM (short username kudu)
I20260620 18:35:26.278299 25762 webserver.cc:460] Webserver started at http://127.10.131.1:38873/ using document root <none> and password file <none>
I20260620 18:35:26.278944 25762 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:26.279054 25762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:26.284403 25762 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:35:26.287479 25811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:26.288915 25762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:35:26.289075 25762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/wal
uuid: "f52b8e69f724413281eff5ef630ed0fa"
format_stamp: "Formatted at 2026-06-20 18:35:20 on dist-test-slave-cbn3"
server_key: "4a5d3f454279ccd50a0dca085831dc67"
server_key_iv: "c9432bfe3a5f3766c2f396fa3bc0b5e2"
server_key_version: "encryptionkey@0"
I20260620 18:35:26.290028 25762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:26.319772 25762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:26.323890 25762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:26.324133 25762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:26.325135 25762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:26.327034 25818 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260620 18:35:26.333029 25762 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 18:35:26.333153 25762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260620 18:35:26.333254 25762 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 18:35:26.336330 25762 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 18:35:26.336431 25762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260620 18:35:26.336803 25818 tablet_bootstrap.cc:492] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa: Bootstrap starting.
I20260620 18:35:26.345319 25818 log.cc:824] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:26.358843 25818 tablet_bootstrap.cc:492] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:35:26.359460 25818 tablet_bootstrap.cc:492] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa: Bootstrap complete.
I20260620 18:35:26.359915 25818 ts_tablet_manager.cc:1397] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa: Time spent bootstrapping tablet: real 0.023s user 0.019s sys 0.000s
I20260620 18:35:26.369190 25818 raft_consensus.cc:348] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f52b8e69f724413281eff5ef630ed0fa" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38513 } } peers { permanent_uuid: "ee6278c92e2b4dcdafaa6356dab839f7" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 42859 } } peers { permanent_uuid: "4696a52c99814944b06cc2a4542d5584" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 45173 } }
I20260620 18:35:26.370555 25818 raft_consensus.cc:729] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f52b8e69f724413281eff5ef630ed0fa, State: Initialized, Role: FOLLOWER
I20260620 18:35:26.371312 25818 consensus_queue.cc:260] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f52b8e69f724413281eff5ef630ed0fa" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38513 } } peers { permanent_uuid: "ee6278c92e2b4dcdafaa6356dab839f7" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 42859 } } peers { permanent_uuid: "4696a52c99814944b06cc2a4542d5584" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 45173 } }
I20260620 18:35:26.372949 25818 ts_tablet_manager.cc:1428] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa: Time spent starting tablet: real 0.013s user 0.016s sys 0.000s
I20260620 18:35:26.385375 25762 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.1:38513
I20260620 18:35:26.385421 25926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.1:38513 every 8 connection(s)
I20260620 18:35:26.386811 25762 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
I20260620 18:35:26.389343 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 25762
I20260620 18:35:26.389966 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.2:42859
--local_ip_for_outbound_sockets=127.10.131.2
--tserver_master_addrs=127.10.131.62:35471
--webserver_port=41945
--webserver_interface=127.10.131.2
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:35:26.391743 25929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:26.389662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.1:49461)
0620 18:35:26.390559 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:26.390859 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":392,"threads_started":1}
W20260620 18:35:26.392333 25927 heartbeater.cc:644] Failed to heartbeat to 127.10.131.62:35471 (0 consecutive failures): Network error: Failed to ping master at 127.10.131.62:35471: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
I20260620 18:35:26.393294 25929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:26.392938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.1:48577)
0620 18:35:26.393079 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:26.393178 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 18:35:26.394325 25929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:26.393968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.1:32999)
0620 18:35:26.394099 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:26.394167 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 18:35:26.394599 25927 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:35:26.726599 25930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:26.726892 25930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:26.727065 25930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:26.734622 25930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:26.734740 25930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:26.734921 25930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.2
I20260620 18:35:26.745025 25930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.2:42859
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.131.2
--webserver_port=41945
--tserver_master_addrs=127.10.131.62:35471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.2
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:35:26.746407 25930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:26.747738 25930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:26.761256 25935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:26.761261 25936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:26.761291 25938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:26.762180 25930 server_base.cc:1034] running on GCE node
I20260620 18:35:26.762790 25930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:26.763689 25930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:26.764945 25930 hybrid_clock.cc:648] HybridClock initialized: now 1781980526764912 us; error 58 us; skew 500 ppm
I20260620 18:35:26.765259 25930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:26 dist-test-slave-cbn3 krb5kdc[24612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980526, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:26.768286 25930 init.cc:377] Logged in from keytab as kudu/127.10.131.2@KRBTEST.COM (short username kudu)
I20260620 18:35:26.769670 25930 webserver.cc:460] Webserver started at http://127.10.131.2:41945/ using document root <none> and password file <none>
I20260620 18:35:26.770320 25930 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:26.770423 25930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:26.775491 25930 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 18:35:26.778067 25945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:26.779417 25930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:35:26.779548 25930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/wal
uuid: "ee6278c92e2b4dcdafaa6356dab839f7"
format_stamp: "Formatted at 2026-06-20 18:35:21 on dist-test-slave-cbn3"
server_key: "ed66ecdb9bad87d9d621d1da087b1b8f"
server_key_iv: "a702c8bf2ed5b0ee0bda99be877c0ffc"
server_key_version: "encryptionkey@0"
I20260620 18:35:26.780447 25930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:26.792012 25930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:26.796285 25930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:26.796509 25930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:26.797480 25930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:26.798698 25930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:26.798800 25930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:26.798877 25930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:26.798930 25930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:26.832370 25930 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.2:42859
I20260620 18:35:26.832479 26057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.2:42859 every 8 connection(s)
I20260620 18:35:26.833979 25930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
I20260620 18:35:26.838784 26060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:26.836925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.2:58157)
0620 18:35:26.837773 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:26.837988 (+ 215us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":329,"threads_started":1}
W20260620 18:35:26.839517 26058 heartbeater.cc:644] Failed to heartbeat to 127.10.131.62:35471 (0 consecutive failures): Network error: Failed to ping master at 127.10.131.62:35471: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
I20260620 18:35:26.840296 26060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:26.839975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.2:58085)
0620 18:35:26.840129 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:26.840195 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 18:35:26.841096 26060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:26.840769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.2:50151)
0620 18:35:26.840916 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:26.840994 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 18:35:26.841324 26058 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 18:35:26.843806 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 25930
I20260620 18:35:26.844424 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.3:45173
--local_ip_for_outbound_sockets=127.10.131.3
--tserver_master_addrs=127.10.131.62:35471
--webserver_port=32829
--webserver_interface=127.10.131.3
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:27.167048 26061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:27.167389 26061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:27.167584 26061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:27.174878 26061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:27.175035 26061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:27.175217 26061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.3
I20260620 18:35:27.185137 26061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.3:45173
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.131.3
--webserver_port=32829
--tserver_master_addrs=127.10.131.62:35471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.3
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:35:27.186510 26061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:27.187848 26061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:27.200763 26066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:27.200825 26069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:27.200803 26067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:27.201643 26061 server_base.cc:1034] running on GCE node
I20260620 18:35:27.202330 26061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:27.203330 26061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:27.204583 26061 hybrid_clock.cc:648] HybridClock initialized: now 1781980527204525 us; error 40 us; skew 500 ppm
I20260620 18:35:27.205003 26061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:27 dist-test-slave-cbn3 krb5kdc[24612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980527, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:27.208168 26061 init.cc:377] Logged in from keytab as kudu/127.10.131.3@KRBTEST.COM (short username kudu)
I20260620 18:35:27.209729 26061 webserver.cc:460] Webserver started at http://127.10.131.3:32829/ using document root <none> and password file <none>
I20260620 18:35:27.210386 26061 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:27.210489 26061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:27.215727 26061 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 18:35:27.218343 26076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:27.219660 26061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:35:27.219856 26061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/wal
uuid: "4696a52c99814944b06cc2a4542d5584"
format_stamp: "Formatted at 2026-06-20 18:35:21 on dist-test-slave-cbn3"
server_key: "4d8aa051697d1eed27322cdb0dd34ffa"
server_key_iv: "f78c6758c8c91cbac28595683e62aaeb"
server_key_version: "encryptionkey@0"
I20260620 18:35:27.220831 26061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:27.233664 26061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:27.237502 26061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:27.237779 26061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:27.238731 26061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:27.239985 26061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:27.240098 26061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:27.240223 26061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:27.240310 26061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:27.273521 26061 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.3:45173
I20260620 18:35:27.273584 26188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.3:45173 every 8 connection(s)
I20260620 18:35:27.274978 26061 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
I20260620 18:35:27.276372 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 26061
I20260620 18:35:27.279908 26192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:27.278182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.3:43227)
0620 18:35:27.278880 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:27.279085 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":270,"threads_started":1}
W20260620 18:35:27.280623 26189 heartbeater.cc:644] Failed to heartbeat to 127.10.131.62:35471 (0 consecutive failures): Network error: Failed to ping master at 127.10.131.62:35471: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
I20260620 18:35:27.281507 26192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:27.281140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.3:55413)
0620 18:35:27.281309 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:27.281379 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 18:35:27.282404 26192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:27.282060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.3:52937)
0620 18:35:27.282209 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:27.282294 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260620 18:35:27.282694 26189 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 18:35:27.395941 26193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:27.395162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.1:45229)
0620 18:35:27.395625 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:27.395737 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":233,"threads_started":1}
I20260620 18:35:27.842602 26194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:27.841815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.2:53575)
0620 18:35:27.842286 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:27.842369 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":220,"threads_started":1}
I20260620 18:35:28.145848 26195 raft_consensus.cc:482] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:35:28.146181 26195 raft_consensus.cc:504] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f52b8e69f724413281eff5ef630ed0fa" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38513 } } peers { permanent_uuid: "ee6278c92e2b4dcdafaa6356dab839f7" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 42859 } } peers { permanent_uuid: "4696a52c99814944b06cc2a4542d5584" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 45173 } }
I20260620 18:35:28.147827 26195 leader_election.cc:288] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee6278c92e2b4dcdafaa6356dab839f7 (127.10.131.2:42859), 4696a52c99814944b06cc2a4542d5584 (127.10.131.3:45173)
Jun 20 18:35:28 dist-test-slave-cbn3 krb5kdc[24612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980526, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for kudu/127.10.131.3@KRBTEST.COM
Jun 20 18:35:28 dist-test-slave-cbn3 krb5kdc[24612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980526, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for kudu/127.10.131.2@KRBTEST.COM
I20260620 18:35:28.175876 26198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:28.149244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:52103 (local address 127.10.131.3:45173)
0620 18:35:28.150052 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:28.150066 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:28.150104 (+ 38us) server_negotiation.cc:407] Connection header received
0620 18:35:28.152334 (+ 2230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:28.152343 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:28.152809 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:28.153223 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:28.156037 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:28.157979 (+ 1942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:28.160925 (+ 2946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:28.161989 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:28.162188 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:28.166967 (+ 4779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:28.166979 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:28.166998 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:28.167046 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:28.171063 (+ 4017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:28.172271 (+ 1208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:28.172280 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:28.172288 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:28.172485 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:28.173058 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:28.173063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:28.173067 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:28.174145 (+ 1078us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:28.174281 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:28.175019 (+ 738us) server_negotiation.cc:299] Negotiation successful
0620 18:35:28.175449 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":659,"threads_started":1}
I20260620 18:35:28.175875 26197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:28.148543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:44569 (local address 127.10.131.2:42859)
0620 18:35:28.149230 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:28.149244 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:28.149296 (+ 52us) server_negotiation.cc:407] Connection header received
0620 18:35:28.152290 (+ 2994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:28.152302 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:28.152810 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:28.153215 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:28.156037 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:28.157979 (+ 1942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:28.160922 (+ 2943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:28.162115 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:28.163041 (+ 926us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:28.167100 (+ 4059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:28.167112 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:28.167128 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:28.167178 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:28.171063 (+ 3885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:28.172271 (+ 1208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:28.172280 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:28.172288 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:28.172485 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:28.173058 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:28.173063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:28.173067 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:28.173798 (+ 731us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:28.173985 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:28.175019 (+ 1034us) server_negotiation.cc:299] Negotiation successful
0620 18:35:28.175449 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":327,"threads_started":1}
I20260620 18:35:28.175913 26199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:28.149498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:45173 (local address 127.10.131.1:52103)
0620 18:35:28.150025 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:28.150043 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:28.150162 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:28.153378 (+ 3216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:28.153390 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:28.154620 (+ 1230us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:28.155810 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:28.155839 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:28.158195 (+ 2356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:28.158199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:28.160749 (+ 2550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:28.160761 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:28.162213 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:28.162221 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:28.162535 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:28.163562 (+ 1027us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:28.163580 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:28.166619 (+ 3039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:28.171237 (+ 4618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:28.171246 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:28.171264 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:28.172115 (+ 851us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:28.172619 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:28.172624 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:28.172628 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:28.172933 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:28.174279 (+ 1346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:28.174284 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:28.174454 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 18:35:28.174765 (+ 311us) client_negotiation.cc:240] Negotiation successful
0620 18:35:28.175471 (+ 706us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":559,"spinlock_wait_cycles":33792,"thread_start_us":263,"threads_started":1}
I20260620 18:35:28.175938 26196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:28.148126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:42859 (local address 127.10.131.1:44569)
0620 18:35:28.148666 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:28.148697 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:28.149341 (+ 644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:28.153378 (+ 4037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:28.153390 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:28.154620 (+ 1230us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:28.155810 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:28.155839 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:28.158191 (+ 2352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:28.158197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:28.160746 (+ 2549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:28.160761 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:28.162356 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:28.162362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:28.162543 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:28.163403 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:28.163426 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:28.166844 (+ 3418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:28.171237 (+ 4393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:28.171246 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:28.171264 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:28.172115 (+ 851us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:28.172619 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:28.172625 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:28.172628 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:28.172933 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:28.173984 (+ 1051us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:28.173991 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:28.174454 (+ 463us) client_negotiation.cc:769] Sending connection context
0620 18:35:28.174765 (+ 311us) client_negotiation.cc:240] Negotiation successful
0620 18:35:28.175352 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":435,"spinlock_wait_cycles":13696,"thread_start_us":269,"threads_started":1}
W20260620 18:35:28.179172 25814 leader_election.cc:341] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4696a52c99814944b06cc2a4542d5584 (127.10.131.3:45173): Not found: Tablet not found: b9d2b93752d9412bb67fe9b77c9144c6
W20260620 18:35:28.179455 25814 leader_election.cc:341] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ee6278c92e2b4dcdafaa6356dab839f7 (127.10.131.2:42859): Not found: Tablet not found: b9d2b93752d9412bb67fe9b77c9144c6
I20260620 18:35:28.179555 25814 leader_election.cc:302] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f52b8e69f724413281eff5ef630ed0fa; no voters: 4696a52c99814944b06cc2a4542d5584, ee6278c92e2b4dcdafaa6356dab839f7
I20260620 18:35:28.179893 26195 raft_consensus.cc:2726] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 18:35:28.284358 26200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:28.283340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.3:54715)
0620 18:35:28.283910 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:28.284112 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":300,"threads_started":1}
I20260620 18:35:28.397331 26199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:28.396804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.1:39483)
0620 18:35:28.397082 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:28.397198 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260620 18:35:28.844409 26201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:28.843599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.2:40585)
0620 18:35:28.844129 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:28.844221 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":280,"threads_started":1}
I20260620 18:35:29.286063 26202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:29.285412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.3:35211)
0620 18:35:29.285829 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:29.285913 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":223,"threads_started":1}
I20260620 18:35:29.399168 26203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:29.398328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.1:42827)
0620 18:35:29.398860 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:29.398973 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":292,"threads_started":1}
I20260620 18:35:29.822858 26204 raft_consensus.cc:482] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:35:29.823091 26204 raft_consensus.cc:504] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f52b8e69f724413281eff5ef630ed0fa" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38513 } } peers { permanent_uuid: "ee6278c92e2b4dcdafaa6356dab839f7" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 42859 } } peers { permanent_uuid: "4696a52c99814944b06cc2a4542d5584" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 45173 } }
I20260620 18:35:29.823745 26204 leader_election.cc:288] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee6278c92e2b4dcdafaa6356dab839f7 (127.10.131.2:42859), 4696a52c99814944b06cc2a4542d5584 (127.10.131.3:45173)
W20260620 18:35:29.824833 25814 leader_election.cc:341] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ee6278c92e2b4dcdafaa6356dab839f7 (127.10.131.2:42859): Not found: Tablet not found: b9d2b93752d9412bb67fe9b77c9144c6
W20260620 18:35:29.825167 25814 leader_election.cc:341] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4696a52c99814944b06cc2a4542d5584 (127.10.131.3:45173): Not found: Tablet not found: b9d2b93752d9412bb67fe9b77c9144c6
I20260620 18:35:29.825287 25814 leader_election.cc:302] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f52b8e69f724413281eff5ef630ed0fa; no voters: 4696a52c99814944b06cc2a4542d5584, ee6278c92e2b4dcdafaa6356dab839f7
I20260620 18:35:29.825570 26204 raft_consensus.cc:2726] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 18:35:29.845948 26205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:29.845290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.2:56197)
0620 18:35:29.845704 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:29.845783 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":214,"threads_started":1}
I20260620 18:35:30.287755 26206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:30.286907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.3:54459)
0620 18:35:30.287442 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:30.287554 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":281,"threads_started":1}
I20260620 18:35:30.401055 26207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:30.400215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.1:58809)
0620 18:35:30.400743 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:30.400828 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":299,"threads_started":1}
I20260620 18:35:30.847493 26208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:30.846859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.2:55419)
0620 18:35:30.847246 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:30.847325 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":192,"threads_started":1}
I20260620 18:35:31.289455 26209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:31.288700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.3:34551)
0620 18:35:31.289180 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:31.289277 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":267,"threads_started":1}
I20260620 18:35:31.402748 26210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:31.401956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.1:39393)
0620 18:35:31.402459 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:31.402553 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":240,"threads_started":1}
I20260620 18:35:31.849403 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:31.848562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.2:45913)
0620 18:35:31.849134 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:31.849226 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":338,"threads_started":1}
I20260620 18:35:32.291158 26212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:32.290445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.3:35925)
0620 18:35:32.290902 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:32.290990 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":221,"threads_started":1}
I20260620 18:35:32.404558 26213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:32.403832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.1:57163)
0620 18:35:32.404290 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:32.404392 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":246,"threads_started":1}
I20260620 18:35:32.462627 26214 raft_consensus.cc:482] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:35:32.462832 26214 raft_consensus.cc:504] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f52b8e69f724413281eff5ef630ed0fa" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38513 } } peers { permanent_uuid: "ee6278c92e2b4dcdafaa6356dab839f7" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 42859 } } peers { permanent_uuid: "4696a52c99814944b06cc2a4542d5584" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 45173 } }
I20260620 18:35:32.463436 26214 leader_election.cc:288] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee6278c92e2b4dcdafaa6356dab839f7 (127.10.131.2:42859), 4696a52c99814944b06cc2a4542d5584 (127.10.131.3:45173)
W20260620 18:35:32.464633 25814 leader_election.cc:341] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ee6278c92e2b4dcdafaa6356dab839f7 (127.10.131.2:42859): Not found: Tablet not found: b9d2b93752d9412bb67fe9b77c9144c6
W20260620 18:35:32.464820 25814 leader_election.cc:341] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4696a52c99814944b06cc2a4542d5584 (127.10.131.3:45173): Not found: Tablet not found: b9d2b93752d9412bb67fe9b77c9144c6
I20260620 18:35:32.464932 25814 leader_election.cc:302] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f52b8e69f724413281eff5ef630ed0fa; no voters: 4696a52c99814944b06cc2a4542d5584, ee6278c92e2b4dcdafaa6356dab839f7
I20260620 18:35:32.465183 26214 raft_consensus.cc:2726] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 18:35:32.851205 26215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:32.850421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.2:38235)
0620 18:35:32.850915 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:32.851004 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":285,"threads_started":1}
I20260620 18:35:33.293084 26216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:33.292160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.3:55799)
0620 18:35:33.292698 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:33.292796 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":279,"threads_started":1}
I20260620 18:35:33.406368 26217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:33.405624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.1:36667)
0620 18:35:33.406090 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:33.406179 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":242,"threads_started":1}
I20260620 18:35:33.853104 26218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:33.852218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.2:41009)
0620 18:35:33.852766 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:33.852899 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":256,"threads_started":1}
I20260620 18:35:34.294900 26219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:34.294109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.3:55041)
0620 18:35:34.294630 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:34.294711 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":212,"threads_started":1}
I20260620 18:35:34.408084 26220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:34.407377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.1:50609)
0620 18:35:34.407848 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:34.407930 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":252,"threads_started":1}
I20260620 18:35:34.854888 26221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:34.854182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.2:46227)
0620 18:35:34.854601 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:34.854688 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":202,"threads_started":1}
I20260620 18:35:35.296540 26222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:35.295829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.3:37673)
0620 18:35:35.296248 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:35.296334 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":208,"threads_started":1}
I20260620 18:35:35.409808 26223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:35.409041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.1:54267)
0620 18:35:35.409554 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:35.409632 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":309,"threads_started":1}
I20260620 18:35:35.856549 26224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:35.855809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.2:51901)
0620 18:35:35.856309 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:35.856392 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":244,"threads_started":1}
I20260620 18:35:35.932008 26225 raft_consensus.cc:482] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:35:35.932265 26225 raft_consensus.cc:504] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f52b8e69f724413281eff5ef630ed0fa" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 38513 } } peers { permanent_uuid: "ee6278c92e2b4dcdafaa6356dab839f7" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 42859 } } peers { permanent_uuid: "4696a52c99814944b06cc2a4542d5584" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 45173 } }
I20260620 18:35:35.933012 26225 leader_election.cc:288] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee6278c92e2b4dcdafaa6356dab839f7 (127.10.131.2:42859), 4696a52c99814944b06cc2a4542d5584 (127.10.131.3:45173)
W20260620 18:35:35.934010 25814 leader_election.cc:341] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4696a52c99814944b06cc2a4542d5584 (127.10.131.3:45173): Not found: Tablet not found: b9d2b93752d9412bb67fe9b77c9144c6
W20260620 18:35:35.934237 25814 leader_election.cc:341] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ee6278c92e2b4dcdafaa6356dab839f7 (127.10.131.2:42859): Not found: Tablet not found: b9d2b93752d9412bb67fe9b77c9144c6
I20260620 18:35:35.934324 25814 leader_election.cc:302] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f52b8e69f724413281eff5ef630ed0fa; no voters: 4696a52c99814944b06cc2a4542d5584, ee6278c92e2b4dcdafaa6356dab839f7
I20260620 18:35:35.934589 26225 raft_consensus.cc:2726] T b9d2b93752d9412bb67fe9b77c9144c6 P f52b8e69f724413281eff5ef630ed0fa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 18:35:36.298171 26226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:36.297479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.3:50095)
0620 18:35:36.297916 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:36.297992 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":209,"threads_started":1}
I20260620 18:35:36.411499 26227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:36.410734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.1:53623)
0620 18:35:36.411214 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:36.411307 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":226,"threads_started":1}
I20260620 18:35:36.858232 26228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:36.857496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.2:37007)
0620 18:35:36.857974 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:36.858066 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":274,"threads_started":1}
I20260620 18:35:37.299734 26229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:37.299084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.3:42995)
0620 18:35:37.299480 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:37.299559 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":194,"threads_started":1}
I20260620 18:35:37.413071 26230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:37.412333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.1:44177)
0620 18:35:37.412744 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:37.412824 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":209,"threads_started":1}
I20260620 18:35:37.859784 26231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:37.859055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.2:33635)
0620 18:35:37.859543 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:37.859623 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":247,"threads_started":1}
I20260620 18:35:38.301440 26232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:38.300663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.3:35365)
0620 18:35:38.301199 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:38.301286 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":320,"threads_started":1}
I20260620 18:35:38.414717 26233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:38.414016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.1:59973)
0620 18:35:38.414448 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:38.414541 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":235,"threads_started":1}
I20260620 18:35:38.861469 26234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:38.860680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.2:34041)
0620 18:35:38.861210 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:38.861303 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":293,"threads_started":1}
I20260620 18:35:39.303124 26235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:39.302380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.3:34559)
0620 18:35:39.302877 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:39.302956 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":266,"threads_started":1}
I20260620 18:35:39.416441 26236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:39.415688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.1:59309)
0620 18:35:39.416129 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:39.416211 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":232,"threads_started":1}
I20260620 18:35:39.863144 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:39.862391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.2:58933)
0620 18:35:39.862856 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:39.862952 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":223,"threads_started":1}
I20260620 18:35:40.304809 26238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:40.304095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.3:37157)
0620 18:35:40.304561 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:40.304650 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":236,"threads_started":1}
I20260620 18:35:40.418167 26239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:40.417478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.1:59987)
0620 18:35:40.417888 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:40.417974 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":225,"threads_started":1}
I20260620 18:35:40.865002 26240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:40.864194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.2:35021)
0620 18:35:40.864676 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:40.864757 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":230,"threads_started":1}
I20260620 18:35:41.306483 26241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:41.305706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.3:60505)
0620 18:35:41.306194 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:41.306289 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":261,"threads_started":1}
I20260620 18:35:41.420096 26242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:41.419352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.1:58271)
0620 18:35:41.419835 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:41.419919 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":242,"threads_started":1}
I20260620 18:35:41.866847 26243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:41.866126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.2:59599)
0620 18:35:41.866569 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:41.866667 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":228,"threads_started":1}
I20260620 18:35:42.281433 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 25762
I20260620 18:35:42.297812 26244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:42.282496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42664 (local address 127.10.131.1:38513)
0620 18:35:42.283136 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:42.283148 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:42.283276 (+ 128us) server_negotiation.cc:407] Connection header received
0620 18:35:42.283534 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:42.283545 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:42.283832 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:42.283963 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:42.286334 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:42.287378 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:42.288698 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:42.289407 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:42.289528 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:42.292141 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:42.292153 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:42.292166 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:42.292202 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:42.295073 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:42.295769 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:42.295780 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:42.295787 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:42.295942 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:42.296323 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:42.296327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:42.296329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:42.296716 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:42.297063 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:42.297315 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 18:35:42.297562 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":244,"threads_started":1}
I20260620 18:35:42.299003 25922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:42.308236 26246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:42.307577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:35471 (local address 127.10.131.3:41867)
0620 18:35:42.307984 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:42.308093 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":205,"threads_started":1}
I20260620 18:35:42.388096 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 25762
I20260620 18:35:42.405464 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 25930
I20260620 18:35:42.418236 26247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:42.406215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50086 (local address 127.10.131.2:42859)
0620 18:35:42.406687 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:42.406691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:42.406727 (+ 36us) server_negotiation.cc:407] Connection header received
0620 18:35:42.406783 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:42.406786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:42.406846 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:42.406935 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:42.408259 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:42.409113 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:42.410237 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:42.410653 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:42.410748 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:42.413279 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:42.413284 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:42.413287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:42.413318 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:42.415892 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:42.416463 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:42.416466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:42.416469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:42.416591 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:42.416950 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:42.416953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:42.416955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:42.417228 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:42.417437 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:42.417697 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 18:35:42.417856 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":261,"threads_started":1}
I20260620 18:35:42.418883 26053 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:42.505009 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 25930
I20260620 18:35:42.523559 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 26061
I20260620 18:35:42.537439 26249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:42.524479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59990 (local address 127.10.131.3:45173)
0620 18:35:42.525170 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:42.525175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:42.525192 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:35:42.525248 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:42.525252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:42.525320 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:42.525411 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:42.527043 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:42.527837 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:42.529207 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:42.529613 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:42.529707 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:42.532240 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:42.532244 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:42.532247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:42.532280 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:42.535062 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:42.535688 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:42.535692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:42.535695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:42.535847 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:42.536195 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:42.536198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:42.536201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:42.536456 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:42.536696 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:42.536944 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 18:35:42.537138 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":374,"threads_started":1}
I20260620 18:35:42.538213 26184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:42.622723 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 26061
2026-06-20T18:35:42Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24190 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:35:42 dist-test-slave-cbn3 krb5kdc[26253](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:35:42 dist-test-slave-cbn3 krb5kdc[26253](info): set up 2 sockets
Jun 20 18:35:42 dist-test-slave-cbn3 krb5kdc[26253](info): commencing operation
krb5kdc: starting...
W20260620 18:35:43.721460 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.011s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:35:43 dist-test-slave-cbn3 krb5kdc[26253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:35:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:35:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.131.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
I20260620 18:35:43.946167 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:38121
--webserver_interface=127.10.131.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:37841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:38121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:44.270876 26266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:44.271211 26266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:44.271359 26266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:44.278553 26266 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 18:35:44.278687 26266 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 18:35:44.278772 26266 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 18:35:44.278860 26266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:35:44.278950 26266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:44.279063 26266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:44.279147 26266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:35:44.279233 26266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:35:44.289139 26266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:37841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:38121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:38121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:35:44.290470 26266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:44.291843 26266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:44.304127 26274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:44.304239 26271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:44.304177 26272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:44.305120 26266 server_base.cc:1034] running on GCE node
I20260620 18:35:44.305872 26266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:44.307236 26266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:44.308490 26266 hybrid_clock.cc:648] HybridClock initialized: now 1781980544308402 us; error 58 us; skew 500 ppm
I20260620 18:35:44.308887 26266 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:44 dist-test-slave-cbn3 krb5kdc[26253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980544, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:44.311964 26266 init.cc:377] Logged in from keytab as kudu/127.10.131.62@KRBTEST.COM (short username kudu)
I20260620 18:35:44.313387 26266 webserver.cc:460] Webserver started at http://127.10.131.62:33489/ using document root <none> and password file <none>
I20260620 18:35:44.314071 26266 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:44.314162 26266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:44.314414 26266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:44.317252 26266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "219191504a5243ac8cc1fcb988930d8c"
format_stamp: "Formatted at 2026-06-20 18:35:44 on dist-test-slave-cbn3"
server_key: "741e9131d0eb1922876e15143dd5e7ef"
server_key_iv: "8e9e30de5f3617c4417b9744be3b87b2"
server_key_version: "encryptionkey@0"
I20260620 18:35:44.317970 26266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "219191504a5243ac8cc1fcb988930d8c"
format_stamp: "Formatted at 2026-06-20 18:35:44 on dist-test-slave-cbn3"
server_key: "741e9131d0eb1922876e15143dd5e7ef"
server_key_iv: "8e9e30de5f3617c4417b9744be3b87b2"
server_key_version: "encryptionkey@0"
I20260620 18:35:44.322932 26266 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 18:35:44.326300 26281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:44.327633 26266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 18:35:44.327827 26266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "219191504a5243ac8cc1fcb988930d8c"
format_stamp: "Formatted at 2026-06-20 18:35:44 on dist-test-slave-cbn3"
server_key: "741e9131d0eb1922876e15143dd5e7ef"
server_key_iv: "8e9e30de5f3617c4417b9744be3b87b2"
server_key_version: "encryptionkey@0"
I20260620 18:35:44.328037 26266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:44.341603 26266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:44.345376 26266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:44.345638 26266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:44.364068 26266 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:38121
I20260620 18:35:44.364169 26332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:38121 every 8 connection(s)
I20260620 18:35:44.365500 26266 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:35:44.368607 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 26266
I20260620 18:35:44.368865 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:35:44.369706 26333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:44.369715 10764 external_mini_cluster.cc:1428] Setting key 5e34bb1bfac13308ad443f3e17ffcdc5
I20260620 18:35:44.384450 26333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c: Bootstrap starting.
Jun 20 18:35:44 dist-test-slave-cbn3 krb5kdc[26253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:35:44.387521 26333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:44.388573 26333 log.cc:824] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:44.391243 26333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c: No bootstrap required, opened a new log
I20260620 18:35:44.393055 26336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:44.371743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42888 (local address 127.10.131.62:38121)
0620 18:35:44.372527 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:44.372538 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:44.372579 (+ 41us) server_negotiation.cc:407] Connection header received
0620 18:35:44.374522 (+ 1943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:44.374542 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:44.374900 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:44.375252 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:44.376979 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:44.378305 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:44.380283 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:44.381026 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:44.381160 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:44.385910 (+ 4750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:44.385917 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:44.385933 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:44.385973 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:44.389077 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:44.389787 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:44.389793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:44.389798 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:44.389943 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:44.390338 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:44.390344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:44.390347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:44.390842 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:44.391141 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:44.391541 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 18:35:44.392020 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":346,"threads_started":1}
I20260620 18:35:44.399484 26333 raft_consensus.cc:348] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219191504a5243ac8cc1fcb988930d8c" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38121 } }
I20260620 18:35:44.399873 26333 raft_consensus.cc:374] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:44.399994 26333 raft_consensus.cc:729] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 219191504a5243ac8cc1fcb988930d8c, State: Initialized, Role: FOLLOWER
I20260620 18:35:44.400570 26333 consensus_queue.cc:260] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219191504a5243ac8cc1fcb988930d8c" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38121 } }
I20260620 18:35:44.400825 26333 raft_consensus.cc:388] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:35:44.401010 26333 raft_consensus.cc:482] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:35:44.401181 26333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:44.403331 26333 raft_consensus.cc:504] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219191504a5243ac8cc1fcb988930d8c" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38121 } }
I20260620 18:35:44.403784 26333 leader_election.cc:302] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 219191504a5243ac8cc1fcb988930d8c; no voters:
I20260620 18:35:44.404376 26333 leader_election.cc:288] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:35:44.404510 26338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:35:44.405290 26338 raft_consensus.cc:686] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [term 1 LEADER]: Becoming Leader. State: Replica: 219191504a5243ac8cc1fcb988930d8c, State: Running, Role: LEADER
I20260620 18:35:44.405831 26338 consensus_queue.cc:237] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219191504a5243ac8cc1fcb988930d8c" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38121 } }
I20260620 18:35:44.406450 26333 sys_catalog.cc:564] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:35:44.408951 26339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "219191504a5243ac8cc1fcb988930d8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219191504a5243ac8cc1fcb988930d8c" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38121 } } }
I20260620 18:35:44.409459 26339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:44.409252 26340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 219191504a5243ac8cc1fcb988930d8c. Latest consensus state: current_term: 1 leader_uuid: "219191504a5243ac8cc1fcb988930d8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219191504a5243ac8cc1fcb988930d8c" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38121 } } }
I20260620 18:35:44.409719 26340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [sys.catalog]: This master's current role is: LEADER
I20260620 18:35:44.410482 26347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:35:44.416484 26347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:35:44.424664 26347 catalog_manager.cc:1347] Generated new cluster ID: b6c54873d26c4dc996c1efb81e4b15b9
I20260620 18:35:44.424769 26347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:35:44.434969 26347 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:35:44.436122 26347 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:35:44.446543 26347 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c: Generated new TSK 0
I20260620 18:35:44.447273 26347 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.131.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab.
Entry for principal kudu/127.10.131.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab.
I20260620 18:35:44.510425 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.1:0
--local_ip_for_outbound_sockets=127.10.131.1
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:38121
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:37841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:44.838274 26359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:44.838534 26359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:44.838711 26359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:44.845937 26359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:44.846064 26359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:44.846237 26359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.1
I20260620 18:35:44.855955 26359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:37841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:38121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.1
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:35:44.857218 26359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:44.858493 26359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:44.871089 26364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:44.871049 26367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:44.871116 26365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:44.871901 26359 server_base.cc:1034] running on GCE node
I20260620 18:35:44.872529 26359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:44.873469 26359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:44.874691 26359 hybrid_clock.cc:648] HybridClock initialized: now 1781980544874648 us; error 52 us; skew 500 ppm
I20260620 18:35:44.874998 26359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:44 dist-test-slave-cbn3 krb5kdc[26253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980544, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:44.878062 26359 init.cc:377] Logged in from keytab as kudu/127.10.131.1@KRBTEST.COM (short username kudu)
I20260620 18:35:44.879451 26359 webserver.cc:460] Webserver started at http://127.10.131.1:37115/ using document root <none> and password file <none>
I20260620 18:35:44.880133 26359 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:44.880275 26359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:44.880578 26359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:44.883682 26359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/data/instance:
uuid: "f28b4329d91c44b98ef80b977708d167"
format_stamp: "Formatted at 2026-06-20 18:35:44 on dist-test-slave-cbn3"
server_key: "5b37b6e9eb0024f30f53b378fd98782f"
server_key_iv: "9c0378c9b550135a00bc9d1cdb6ee6dd"
server_key_version: "encryptionkey@0"
I20260620 18:35:44.884399 26359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance:
uuid: "f28b4329d91c44b98ef80b977708d167"
format_stamp: "Formatted at 2026-06-20 18:35:44 on dist-test-slave-cbn3"
server_key: "5b37b6e9eb0024f30f53b378fd98782f"
server_key_iv: "9c0378c9b550135a00bc9d1cdb6ee6dd"
server_key_version: "encryptionkey@0"
I20260620 18:35:44.889307 26359 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 18:35:44.892782 26374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:44.894063 26359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:35:44.894230 26359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/wal
uuid: "f28b4329d91c44b98ef80b977708d167"
format_stamp: "Formatted at 2026-06-20 18:35:44 on dist-test-slave-cbn3"
server_key: "5b37b6e9eb0024f30f53b378fd98782f"
server_key_iv: "9c0378c9b550135a00bc9d1cdb6ee6dd"
server_key_version: "encryptionkey@0"
I20260620 18:35:44.894418 26359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:44.911612 26359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:44.915594 26359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:44.915877 26359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:44.916790 26359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:44.918236 26359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:44.918331 26359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:44.918474 26359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:44.918542 26359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:44.952196 26359 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.1:40323
I20260620 18:35:44.952232 26486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.1:40323 every 8 connection(s)
I20260620 18:35:44.953743 26359 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
I20260620 18:35:44.963716 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 26359
I20260620 18:35:44.963992 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance
I20260620 18:35:44.964932 10764 external_mini_cluster.cc:1428] Setting key 711d9cc3c12a0ed925799952d7b25205
Jun 20 18:35:44 dist-test-slave-cbn3 krb5kdc[26253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980544, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:35:44.975660 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:44.956291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:32825 (local address 127.10.131.62:38121)
0620 18:35:44.956756 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:44.956761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:44.957813 (+ 1052us) server_negotiation.cc:407] Connection header received
0620 18:35:44.959851 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:44.959857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:44.959982 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:44.960110 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:44.961871 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:44.962650 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:44.965138 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:44.965643 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:44.965795 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:44.970033 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:44.970037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:44.970041 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:44.970080 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:44.972585 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:44.973372 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:44.973377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:44.973385 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:44.973496 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:44.973924 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:44.973929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:44.973932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:44.974197 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:44.974351 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:44.975049 (+ 698us) server_negotiation.cc:299] Negotiation successful
0620 18:35:44.975292 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":238,"threads_started":1}
I20260620 18:35:44.976665 26490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:44.956879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.1:32825)
0620 18:35:44.957632 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:44.957683 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:44.959532 (+ 1849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:44.960412 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:44.960421 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:44.961063 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:44.961684 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:44.961702 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:44.962849 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:44.962853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:44.964978 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:44.964999 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:44.965958 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:44.965967 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:44.966243 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:44.967057 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:44.967088 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:44.969796 (+ 2708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:44.972725 (+ 2929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:44.972734 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:44.972753 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:44.973248 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:44.973608 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:44.973613 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:44.973617 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:44.973807 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:44.974322 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:44.974331 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:44.974684 (+ 353us) client_negotiation.cc:769] Sending connection context
0620 18:35:44.974947 (+ 263us) client_negotiation.cc:240] Negotiation successful
0620 18:35:44.975474 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":273,"threads_started":1}
I20260620 18:35:44.978204 26487 heartbeater.cc:347] Connected to a master server at 127.10.131.62:38121
I20260620 18:35:44.978535 26487 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:44.979426 26487 heartbeater.cc:511] Master 127.10.131.62:38121 requested a full tablet report, sending...
I20260620 18:35:44.981662 26298 ts_manager.cc:194] Registered new tserver with Master: f28b4329d91c44b98ef80b977708d167 (127.10.131.1:40323)
I20260620 18:35:44.983094 26298 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.1@KRBTEST.COM'} at 127.10.131.1:32825
WARNING: no policy specified for kudu/127.10.131.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab.
Entry for principal kudu/127.10.131.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab.
I20260620 18:35:45.016314 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.2:0
--local_ip_for_outbound_sockets=127.10.131.2
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:38121
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:37841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:45.342094 26493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:45.342409 26493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:45.342501 26493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:45.349823 26493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:45.349922 26493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:45.350031 26493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.2
I20260620 18:35:45.359688 26493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:37841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:38121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.2
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:35:45.360903 26493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:45.362150 26493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:45.375336 26499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:45.375545 26501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:45.375947 26498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:45.376919 26493 server_base.cc:1034] running on GCE node
I20260620 18:35:45.377624 26493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:45.378623 26493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:45.379920 26493 hybrid_clock.cc:648] HybridClock initialized: now 1781980545379849 us; error 49 us; skew 500 ppm
I20260620 18:35:45.380286 26493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:45 dist-test-slave-cbn3 krb5kdc[26253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980545, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:45.383760 26493 init.cc:377] Logged in from keytab as kudu/127.10.131.2@KRBTEST.COM (short username kudu)
I20260620 18:35:45.385332 26493 webserver.cc:460] Webserver started at http://127.10.131.2:33655/ using document root <none> and password file <none>
I20260620 18:35:45.386020 26493 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:45.386168 26493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:45.386500 26493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:45.389540 26493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/data/instance:
uuid: "41e3a33ef4884818ab9abd62b74412ac"
format_stamp: "Formatted at 2026-06-20 18:35:45 on dist-test-slave-cbn3"
server_key: "5b944fe391bc39257d5882455852ec03"
server_key_iv: "6268bea237db663cdf38df60554e3ab5"
server_key_version: "encryptionkey@0"
I20260620 18:35:45.390357 26493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance:
uuid: "41e3a33ef4884818ab9abd62b74412ac"
format_stamp: "Formatted at 2026-06-20 18:35:45 on dist-test-slave-cbn3"
server_key: "5b944fe391bc39257d5882455852ec03"
server_key_iv: "6268bea237db663cdf38df60554e3ab5"
server_key_version: "encryptionkey@0"
I20260620 18:35:45.395484 26493 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:35:45.398945 26508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:45.400149 26493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:35:45.400348 26493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/wal
uuid: "41e3a33ef4884818ab9abd62b74412ac"
format_stamp: "Formatted at 2026-06-20 18:35:45 on dist-test-slave-cbn3"
server_key: "5b944fe391bc39257d5882455852ec03"
server_key_iv: "6268bea237db663cdf38df60554e3ab5"
server_key_version: "encryptionkey@0"
I20260620 18:35:45.400530 26493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:45.418082 26493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:45.422591 26493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:45.422926 26493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:45.423942 26493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:45.425357 26493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:45.425455 26493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:45.425598 26493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:45.425668 26493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:45.458819 26493 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.2:40545
I20260620 18:35:45.458899 26620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.2:40545 every 8 connection(s)
I20260620 18:35:45.460095 26493 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
I20260620 18:35:45.468746 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 26493
I20260620 18:35:45.468969 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance
I20260620 18:35:45.469559 10764 external_mini_cluster.cc:1428] Setting key 71be65c9bb96130f5772a86f7278c629
Jun 20 18:35:45 dist-test-slave-cbn3 krb5kdc[26253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980545, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:35:45.480453 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:45.462726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:41129 (local address 127.10.131.62:38121)
0620 18:35:45.462905 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:45.462910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:45.463991 (+ 1081us) server_negotiation.cc:407] Connection header received
0620 18:35:45.465799 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:45.465805 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:45.465890 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:45.465997 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:45.467435 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:45.468148 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:45.470344 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:45.470912 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:45.471030 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:45.474759 (+ 3729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:45.474766 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:45.474770 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:45.474805 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:45.477453 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:45.478179 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:45.478189 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:45.478192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:45.478320 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:45.478738 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:45.478742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:45.478745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:45.479011 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:45.479157 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:45.479870 (+ 713us) server_negotiation.cc:299] Negotiation successful
0620 18:35:45.480106 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:35:45.481499 26623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:45.463178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.2:41129)
0620 18:35:45.463833 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:45.463872 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:45.465546 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:45.466203 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:45.466212 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:45.466735 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:45.467273 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:45.467289 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:45.468312 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:45.468316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:45.470208 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:45.470219 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:45.471104 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:45.471109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:45.471389 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:45.472110 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:45.472135 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:45.474540 (+ 2405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:45.477595 (+ 3055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:45.477604 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:45.477621 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:45.478053 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:45.478439 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:45.478445 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:45.478449 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:45.478629 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:45.479130 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:45.479137 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:45.479506 (+ 369us) client_negotiation.cc:769] Sending connection context
0620 18:35:45.479784 (+ 278us) client_negotiation.cc:240] Negotiation successful
0620 18:35:45.480312 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":244,"threads_started":1}
I20260620 18:35:45.482935 26621 heartbeater.cc:347] Connected to a master server at 127.10.131.62:38121
I20260620 18:35:45.483251 26621 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:45.484119 26621 heartbeater.cc:511] Master 127.10.131.62:38121 requested a full tablet report, sending...
I20260620 18:35:45.485764 26298 ts_manager.cc:194] Registered new tserver with Master: 41e3a33ef4884818ab9abd62b74412ac (127.10.131.2:40545)
I20260620 18:35:45.486969 26298 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.2@KRBTEST.COM'} at 127.10.131.2:41129
WARNING: no policy specified for kudu/127.10.131.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab.
Entry for principal kudu/127.10.131.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab.
I20260620 18:35:45.520841 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.3:0
--local_ip_for_outbound_sockets=127.10.131.3
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:38121
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:37841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:35:45.853159 26626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:35:45.853425 26626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:35:45.853598 26626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:35:45.860888 26626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:35:45.861013 26626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:35:45.861198 26626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.3
I20260620 18:35:45.870663 26626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:37841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:38121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.3
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:35:45.871917 26626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:35:45.873308 26626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:35:45.886487 26634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:45.886757 26631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:35:45.886718 26632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:35:45.887634 26626 server_base.cc:1034] running on GCE node
I20260620 18:35:45.888200 26626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:35:45.889242 26626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:35:45.890458 26626 hybrid_clock.cc:648] HybridClock initialized: now 1781980545890404 us; error 56 us; skew 500 ppm
I20260620 18:35:45.890761 26626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:35:45 dist-test-slave-cbn3 krb5kdc[26253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980545, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:35:45.894073 26626 init.cc:377] Logged in from keytab as kudu/127.10.131.3@KRBTEST.COM (short username kudu)
I20260620 18:35:45.895406 26626 webserver.cc:460] Webserver started at http://127.10.131.3:44637/ using document root <none> and password file <none>
I20260620 18:35:45.895998 26626 fs_manager.cc:359] Metadata directory not provided
I20260620 18:35:45.896101 26626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:35:45.896346 26626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:35:45.899421 26626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/data/instance:
uuid: "e0201587fac944f7a411a55384847506"
format_stamp: "Formatted at 2026-06-20 18:35:45 on dist-test-slave-cbn3"
server_key: "0c42b0ce96c7bc82cae239b8700e7d1a"
server_key_iv: "4a2bcd7a747c20d3e1a8653578c680d5"
server_key_version: "encryptionkey@0"
I20260620 18:35:45.900139 26626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance:
uuid: "e0201587fac944f7a411a55384847506"
format_stamp: "Formatted at 2026-06-20 18:35:45 on dist-test-slave-cbn3"
server_key: "0c42b0ce96c7bc82cae239b8700e7d1a"
server_key_iv: "4a2bcd7a747c20d3e1a8653578c680d5"
server_key_version: "encryptionkey@0"
I20260620 18:35:45.905428 26626 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 18:35:45.909004 26641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:45.910334 26626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:35:45.910499 26626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/wal
uuid: "e0201587fac944f7a411a55384847506"
format_stamp: "Formatted at 2026-06-20 18:35:45 on dist-test-slave-cbn3"
server_key: "0c42b0ce96c7bc82cae239b8700e7d1a"
server_key_iv: "4a2bcd7a747c20d3e1a8653578c680d5"
server_key_version: "encryptionkey@0"
I20260620 18:35:45.910682 26626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:35:45.927070 26626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:35:45.930987 26626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:35:45.931249 26626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:35:45.932325 26626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:35:45.933660 26626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:35:45.933730 26626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:45.933866 26626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:35:45.933928 26626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:35:45.968551 26626 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.3:33301
I20260620 18:35:45.968595 26753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.3:33301 every 8 connection(s)
I20260620 18:35:45.969885 26626 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
I20260620 18:35:45.972580 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 26626
I20260620 18:35:45.972894 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance
I20260620 18:35:45.973402 10764 external_mini_cluster.cc:1428] Setting key 26689ae4bced96a8e0c813925a245730
Jun 20 18:35:45 dist-test-slave-cbn3 krb5kdc[26253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980545, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:35:45.986842 26487 heartbeater.cc:503] Master 127.10.131.62:38121 was elected leader, sending a full tablet report...
I20260620 18:35:45.990320 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:45.972587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:40267 (local address 127.10.131.62:38121)
0620 18:35:45.972892 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:45.972898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:45.974170 (+ 1272us) server_negotiation.cc:407] Connection header received
0620 18:35:45.976053 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:45.976059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:45.976157 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:45.976271 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:45.977850 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:45.978682 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:45.980275 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:45.980721 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:45.980886 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:45.984271 (+ 3385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:45.984276 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:45.984279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:45.984311 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:45.987002 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:45.987899 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:45.987906 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:45.987913 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:45.988107 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:45.988632 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:45.988637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:45.988639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:45.988961 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:45.989110 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:45.989754 (+ 644us) server_negotiation.cc:299] Negotiation successful
0620 18:35:45.989958 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260620 18:35:45.991148 26756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:45.973143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.3:40267)
0620 18:35:45.973996 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:45.974049 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:45.975776 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:45.976503 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:45.976511 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:45.977124 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:35:45.977680 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:45.977697 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:45.978850 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:45.978854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:45.980117 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:45.980130 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:45.980968 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:45.980990 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:45.981233 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:45.981902 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:35:45.981925 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:35:45.984068 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:35:45.987184 (+ 3116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:45.987193 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:45.987207 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:45.987758 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:45.988240 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:35:45.988245 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:35:45.988250 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:35:45.988481 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:35:45.989092 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:35:45.989099 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:35:45.989418 (+ 319us) client_negotiation.cc:769] Sending connection context
0620 18:35:45.989654 (+ 236us) client_negotiation.cc:240] Negotiation successful
0620 18:35:45.990160 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":352,"threads_started":1}
I20260620 18:35:45.992473 26754 heartbeater.cc:347] Connected to a master server at 127.10.131.62:38121
I20260620 18:35:45.992758 26754 heartbeater.cc:464] Registering TS with master...
I20260620 18:35:45.993466 26754 heartbeater.cc:511] Master 127.10.131.62:38121 requested a full tablet report, sending...
I20260620 18:35:45.994755 26297 ts_manager.cc:194] Registered new tserver with Master: e0201587fac944f7a411a55384847506 (127.10.131.3:33301)
I20260620 18:35:45.995576 26297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.3@KRBTEST.COM'} at 127.10.131.3:40267
I20260620 18:35:46.000665 10764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:35:46.018567 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:46.005181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42894 (local address 127.10.131.62:38121)
0620 18:35:46.005381 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:46.005389 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:46.005786 (+ 397us) server_negotiation.cc:407] Connection header received
0620 18:35:46.005896 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:46.005900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:46.005960 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:46.006045 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:46.007641 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.008378 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.009922 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.010330 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.010421 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:46.013110 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:46.013115 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:46.013118 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:46.013147 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:46.015900 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:46.016585 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:46.016588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:46.016591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:46.016714 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:46.017315 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:46.017318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:46.017320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:46.017628 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:46.017816 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:46.018084 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 18:35:46.018245 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:35:46.022493 26297 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:35:46.029382 26297 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:35:46.054160 26768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:46.043952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:40323 (local address 127.0.0.1:59092)
0620 18:35:46.045167 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:46.045180 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:46.045305 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:46.046170 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:46.046174 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:46.046198 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:46.046667 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:46.046679 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.048594 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.048599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:46.050705 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:46.050719 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.052664 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.052669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:46.053237 (+ 568us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:46.053392 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 18:35:46.053532 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 18:35:46.053727 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1122,"mutex_wait_us":128,"thread_start_us":1547,"threads_started":1}
I20260620 18:35:46.054203 26767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:46.042642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34198 (local address 127.10.131.2:40545)
0620 18:35:46.044579 (+ 1937us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:46.044592 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:46.044627 (+ 35us) server_negotiation.cc:407] Connection header received
0620 18:35:46.044759 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:46.044770 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:46.045081 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:46.045291 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:46.046006 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.047739 (+ 1733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.050648 (+ 2909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.052135 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.052282 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:46.052479 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:46.053701 (+ 1222us) server_negotiation.cc:299] Negotiation successful
0620 18:35:46.053872 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1740,"thread_start_us":1588,"threads_started":1}
I20260620 18:35:46.055881 26769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:46.044125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59092 (local address 127.10.131.1:40323)
0620 18:35:46.044974 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:46.044985 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:46.045419 (+ 434us) server_negotiation.cc:407] Connection header received
0620 18:35:46.045518 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:46.045537 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:46.045753 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:46.045928 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:46.046878 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.048416 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.050867 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.052195 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.052306 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:46.052517 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:46.055400 (+ 2883us) server_negotiation.cc:299] Negotiation successful
0620 18:35:46.055600 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":510,"threads_started":1}
I20260620 18:35:46.055258 26766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:46.042385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:40545 (local address 127.0.0.1:34198)
0620 18:35:46.043184 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:46.043241 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:46.043451 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:46.045297 (+ 1846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:46.045303 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:46.045337 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:46.045818 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:46.045833 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.047902 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.047907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:46.050441 (+ 2534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:46.050455 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.052636 (+ 2181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.052648 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:46.053264 (+ 616us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:46.053392 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 18:35:46.053522 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 18:35:46.054790 (+ 1268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":1408,"thread_start_us":335,"threads_started":1}
I20260620 18:35:46.060259 26422 tablet_service.cc:1467] Processing CreateTablet for tablet 3a2ea5623bed47b2837dcb3f4bda17b4 (DEFAULT_TABLE table=test-table [id=8d071f02d2614a808c4097d45b43496a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:46.060194 26556 tablet_service.cc:1467] Processing CreateTablet for tablet 3a2ea5623bed47b2837dcb3f4bda17b4 (DEFAULT_TABLE table=test-table [id=8d071f02d2614a808c4097d45b43496a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:46.061517 26422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a2ea5623bed47b2837dcb3f4bda17b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:46.061611 26556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a2ea5623bed47b2837dcb3f4bda17b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:46.065207 26768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:46.045852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:33301 (local address 127.0.0.1:55994)
0620 18:35:46.056218 (+ 10366us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:46.056233 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:46.056320 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:46.058346 (+ 2026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:46.058351 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:46.058382 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:46.058567 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:46.058578 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.060453 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.060459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:46.062313 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:46.062327 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.064056 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.064071 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:46.064482 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:46.064519 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 18:35:46.064626 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 18:35:46.064769 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10268,"thread_start_us":10835,"threads_started":1}
I20260620 18:35:46.065987 26771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:46.046477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55994 (local address 127.10.131.3:33301)
0620 18:35:46.057548 (+ 11071us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:46.057579 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:46.057616 (+ 37us) server_negotiation.cc:407] Connection header received
0620 18:35:46.057719 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:46.057727 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:46.057984 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:46.058188 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:46.058753 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.060301 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.062485 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.063887 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.064018 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:46.064245 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:46.065494 (+ 1249us) server_negotiation.cc:299] Negotiation successful
0620 18:35:46.065662 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10898,"thread_start_us":10753,"threads_started":1}
I20260620 18:35:46.070132 26689 tablet_service.cc:1467] Processing CreateTablet for tablet 3a2ea5623bed47b2837dcb3f4bda17b4 (DEFAULT_TABLE table=test-table [id=8d071f02d2614a808c4097d45b43496a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:35:46.071667 26689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a2ea5623bed47b2837dcb3f4bda17b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:35:46.073930 26772 tablet_bootstrap.cc:492] T 3a2ea5623bed47b2837dcb3f4bda17b4 P 41e3a33ef4884818ab9abd62b74412ac: Bootstrap starting.
I20260620 18:35:46.077103 26772 tablet_bootstrap.cc:654] T 3a2ea5623bed47b2837dcb3f4bda17b4 P 41e3a33ef4884818ab9abd62b74412ac: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:46.077584 26773 tablet_bootstrap.cc:492] T 3a2ea5623bed47b2837dcb3f4bda17b4 P f28b4329d91c44b98ef80b977708d167: Bootstrap starting.
I20260620 18:35:46.078466 26772 log.cc:824] T 3a2ea5623bed47b2837dcb3f4bda17b4 P 41e3a33ef4884818ab9abd62b74412ac: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:46.081200 26773 tablet_bootstrap.cc:654] T 3a2ea5623bed47b2837dcb3f4bda17b4 P f28b4329d91c44b98ef80b977708d167: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:46.082504 26773 log.cc:824] T 3a2ea5623bed47b2837dcb3f4bda17b4 P f28b4329d91c44b98ef80b977708d167: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:46.082728 26772 tablet_bootstrap.cc:492] T 3a2ea5623bed47b2837dcb3f4bda17b4 P 41e3a33ef4884818ab9abd62b74412ac: No bootstrap required, opened a new log
I20260620 18:35:46.083009 26772 ts_tablet_manager.cc:1397] T 3a2ea5623bed47b2837dcb3f4bda17b4 P 41e3a33ef4884818ab9abd62b74412ac: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20260620 18:35:46.085518 26773 tablet_bootstrap.cc:492] T 3a2ea5623bed47b2837dcb3f4bda17b4 P f28b4329d91c44b98ef80b977708d167: No bootstrap required, opened a new log
I20260620 18:35:46.085780 26773 ts_tablet_manager.cc:1397] T 3a2ea5623bed47b2837dcb3f4bda17b4 P f28b4329d91c44b98ef80b977708d167: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260620 18:35:46.089574 26776 tablet_bootstrap.cc:492] T 3a2ea5623bed47b2837dcb3f4bda17b4 P e0201587fac944f7a411a55384847506: Bootstrap starting.
I20260620 18:35:46.093786 26776 tablet_bootstrap.cc:654] T 3a2ea5623bed47b2837dcb3f4bda17b4 P e0201587fac944f7a411a55384847506: Neither blocks nor log segments found. Creating new log.
I20260620 18:35:46.094007 26772 raft_consensus.cc:348] T 3a2ea5623bed47b2837dcb3f4bda17b4 P 41e3a33ef4884818ab9abd62b74412ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41e3a33ef4884818ab9abd62b74412ac" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40545 } } peers { permanent_uuid: "f28b4329d91c44b98ef80b977708d167" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40323 } } peers { permanent_uuid: "e0201587fac944f7a411a55384847506" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 33301 } }
I20260620 18:35:46.094589 26772 raft_consensus.cc:374] T 3a2ea5623bed47b2837dcb3f4bda17b4 P 41e3a33ef4884818ab9abd62b74412ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:46.094735 26772 raft_consensus.cc:729] T 3a2ea5623bed47b2837dcb3f4bda17b4 P 41e3a33ef4884818ab9abd62b74412ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41e3a33ef4884818ab9abd62b74412ac, State: Initialized, Role: FOLLOWER
I20260620 18:35:46.095191 26776 log.cc:824] T 3a2ea5623bed47b2837dcb3f4bda17b4 P e0201587fac944f7a411a55384847506: Log is configured to *not* fsync() on all Append() calls
I20260620 18:35:46.095348 26772 consensus_queue.cc:260] T 3a2ea5623bed47b2837dcb3f4bda17b4 P 41e3a33ef4884818ab9abd62b74412ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41e3a33ef4884818ab9abd62b74412ac" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40545 } } peers { permanent_uuid: "f28b4329d91c44b98ef80b977708d167" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40323 } } peers { permanent_uuid: "e0201587fac944f7a411a55384847506" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 33301 } }
I20260620 18:35:46.096395 26773 raft_consensus.cc:348] T 3a2ea5623bed47b2837dcb3f4bda17b4 P f28b4329d91c44b98ef80b977708d167 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41e3a33ef4884818ab9abd62b74412ac" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40545 } } peers { permanent_uuid: "f28b4329d91c44b98ef80b977708d167" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40323 } } peers { permanent_uuid: "e0201587fac944f7a411a55384847506" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 33301 } }
I20260620 18:35:46.097066 26773 raft_consensus.cc:374] T 3a2ea5623bed47b2837dcb3f4bda17b4 P f28b4329d91c44b98ef80b977708d167 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:46.097193 26773 raft_consensus.cc:729] T 3a2ea5623bed47b2837dcb3f4bda17b4 P f28b4329d91c44b98ef80b977708d167 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f28b4329d91c44b98ef80b977708d167, State: Initialized, Role: FOLLOWER
I20260620 18:35:46.097939 26773 consensus_queue.cc:260] T 3a2ea5623bed47b2837dcb3f4bda17b4 P f28b4329d91c44b98ef80b977708d167 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41e3a33ef4884818ab9abd62b74412ac" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40545 } } peers { permanent_uuid: "f28b4329d91c44b98ef80b977708d167" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40323 } } peers { permanent_uuid: "e0201587fac944f7a411a55384847506" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 33301 } }
I20260620 18:35:46.100369 26772 ts_tablet_manager.cc:1428] T 3a2ea5623bed47b2837dcb3f4bda17b4 P 41e3a33ef4884818ab9abd62b74412ac: Time spent starting tablet: real 0.017s user 0.009s sys 0.006s
I20260620 18:35:46.100771 26621 heartbeater.cc:503] Master 127.10.131.62:38121 was elected leader, sending a full tablet report...
I20260620 18:35:46.102032 26773 ts_tablet_manager.cc:1428] T 3a2ea5623bed47b2837dcb3f4bda17b4 P f28b4329d91c44b98ef80b977708d167: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260620 18:35:46.105456 26776 tablet_bootstrap.cc:492] T 3a2ea5623bed47b2837dcb3f4bda17b4 P e0201587fac944f7a411a55384847506: No bootstrap required, opened a new log
I20260620 18:35:46.105762 26776 ts_tablet_manager.cc:1397] T 3a2ea5623bed47b2837dcb3f4bda17b4 P e0201587fac944f7a411a55384847506: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.004s
I20260620 18:35:46.113212 26776 raft_consensus.cc:348] T 3a2ea5623bed47b2837dcb3f4bda17b4 P e0201587fac944f7a411a55384847506 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41e3a33ef4884818ab9abd62b74412ac" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40545 } } peers { permanent_uuid: "f28b4329d91c44b98ef80b977708d167" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40323 } } peers { permanent_uuid: "e0201587fac944f7a411a55384847506" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 33301 } }
I20260620 18:35:46.113669 26776 raft_consensus.cc:374] T 3a2ea5623bed47b2837dcb3f4bda17b4 P e0201587fac944f7a411a55384847506 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:35:46.113778 26776 raft_consensus.cc:729] T 3a2ea5623bed47b2837dcb3f4bda17b4 P e0201587fac944f7a411a55384847506 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0201587fac944f7a411a55384847506, State: Initialized, Role: FOLLOWER
I20260620 18:35:46.114372 26776 consensus_queue.cc:260] T 3a2ea5623bed47b2837dcb3f4bda17b4 P e0201587fac944f7a411a55384847506 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41e3a33ef4884818ab9abd62b74412ac" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40545 } } peers { permanent_uuid: "f28b4329d91c44b98ef80b977708d167" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40323 } } peers { permanent_uuid: "e0201587fac944f7a411a55384847506" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 33301 } }
I20260620 18:35:46.115531 26754 heartbeater.cc:503] Master 127.10.131.62:38121 was elected leader, sending a full tablet report...
I20260620 18:35:46.116127 26776 ts_tablet_manager.cc:1428] T 3a2ea5623bed47b2837dcb3f4bda17b4 P e0201587fac944f7a411a55384847506: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260620 18:35:46.161163 26778 raft_consensus.cc:482] T 3a2ea5623bed47b2837dcb3f4bda17b4 P 41e3a33ef4884818ab9abd62b74412ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:35:46.161553 26778 raft_consensus.cc:504] T 3a2ea5623bed47b2837dcb3f4bda17b4 P 41e3a33ef4884818ab9abd62b74412ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41e3a33ef4884818ab9abd62b74412ac" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40545 } } peers { permanent_uuid: "f28b4329d91c44b98ef80b977708d167" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40323 } } peers { permanent_uuid: "e0201587fac944f7a411a55384847506" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 33301 } }
I20260620 18:35:46.163422 26778 leader_election.cc:288] T 3a2ea5623bed47b2837dcb3f4bda17b4 P 41e3a33ef4884818ab9abd62b74412ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f28b4329d91c44b98ef80b977708d167 (127.10.131.1:40323), e0201587fac944f7a411a55384847506 (127.10.131.3:33301)
I20260620 18:35:46.171123 26769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:46.163936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:40199 (local address 127.10.131.1:40323)
0620 18:35:46.164182 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:46.164186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:46.164764 (+ 578us) server_negotiation.cc:407] Connection header received
0620 18:35:46.164989 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:46.164995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:46.165088 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:46.165232 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:46.165860 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.166861 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.169017 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.170108 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.170239 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:46.170288 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:46.170690 (+ 402us) server_negotiation.cc:299] Negotiation successful
0620 18:35:46.170866 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 18:35:46.171198 26781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:46.163839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:40323 (local address 127.10.131.2:40199)
0620 18:35:46.164655 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:46.164674 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:46.164812 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:46.165260 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:46.165265 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:46.165307 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:46.165683 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:46.165692 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.167035 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.167041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:46.168829 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:46.168881 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.170327 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.170335 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:46.170529 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:46.170564 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 18:35:46.170665 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 18:35:46.170797 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":445,"threads_started":1}
I20260620 18:35:46.171939 26771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:46.164980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:45123 (local address 127.10.131.3:33301)
0620 18:35:46.165204 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:46.165210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:46.165483 (+ 273us) server_negotiation.cc:407] Connection header received
0620 18:35:46.165677 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:46.165683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:46.165746 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:46.165837 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:35:46.166662 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.167699 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.169726 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.170703 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.170829 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:46.170893 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:46.171564 (+ 671us) server_negotiation.cc:299] Negotiation successful
0620 18:35:46.171699 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 18:35:46.171960 26782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:46.164750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:33301 (local address 127.10.131.2:45123)
0620 18:35:46.165391 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0620 18:35:46.165405 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:35:46.165551 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:35:46.166088 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:35:46.166091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:35:46.166109 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:35:46.166467 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:46.166486 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.167848 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.167853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:46.169580 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:35:46.169594 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.171054 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.171059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:35:46.171255 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:46.171438 (+ 183us) client_negotiation.cc:769] Sending connection context
0620 18:35:46.171526 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 18:35:46.171657 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":460,"threads_started":1}
I20260620 18:35:46.172188 26442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a2ea5623bed47b2837dcb3f4bda17b4" candidate_uuid: "41e3a33ef4884818ab9abd62b74412ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f28b4329d91c44b98ef80b977708d167" is_pre_election: true
I20260620 18:35:46.172808 26442 raft_consensus.cc:2454] T 3a2ea5623bed47b2837dcb3f4bda17b4 P f28b4329d91c44b98ef80b977708d167 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41e3a33ef4884818ab9abd62b74412ac in term 0.
I20260620 18:35:46.172991 26709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a2ea5623bed47b2837dcb3f4bda17b4" candidate_uuid: "41e3a33ef4884818ab9abd62b74412ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0201587fac944f7a411a55384847506" is_pre_election: true
I20260620 18:35:46.173483 26709 raft_consensus.cc:2454] T 3a2ea5623bed47b2837dcb3f4bda17b4 P e0201587fac944f7a411a55384847506 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41e3a33ef4884818ab9abd62b74412ac in term 0.
I20260620 18:35:46.173561 26509 leader_election.cc:302] T 3a2ea5623bed47b2837dcb3f4bda17b4 P 41e3a33ef4884818ab9abd62b74412ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41e3a33ef4884818ab9abd62b74412ac, f28b4329d91c44b98ef80b977708d167; no voters:
I20260620 18:35:46.173969 26778 raft_consensus.cc:2781] T 3a2ea5623bed47b2837dcb3f4bda17b4 P 41e3a33ef4884818ab9abd62b74412ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:35:46.174113 26778 raft_consensus.cc:482] T 3a2ea5623bed47b2837dcb3f4bda17b4 P 41e3a33ef4884818ab9abd62b74412ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:35:46.174222 26778 raft_consensus.cc:3037] T 3a2ea5623bed47b2837dcb3f4bda17b4 P 41e3a33ef4884818ab9abd62b74412ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:46.176383 26778 raft_consensus.cc:504] T 3a2ea5623bed47b2837dcb3f4bda17b4 P 41e3a33ef4884818ab9abd62b74412ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41e3a33ef4884818ab9abd62b74412ac" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40545 } } peers { permanent_uuid: "f28b4329d91c44b98ef80b977708d167" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40323 } } peers { permanent_uuid: "e0201587fac944f7a411a55384847506" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 33301 } }
I20260620 18:35:46.177026 26778 leader_election.cc:288] T 3a2ea5623bed47b2837dcb3f4bda17b4 P 41e3a33ef4884818ab9abd62b74412ac [CANDIDATE]: Term 1 election: Requested vote from peers f28b4329d91c44b98ef80b977708d167 (127.10.131.1:40323), e0201587fac944f7a411a55384847506 (127.10.131.3:33301)
I20260620 18:35:46.177357 26442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a2ea5623bed47b2837dcb3f4bda17b4" candidate_uuid: "41e3a33ef4884818ab9abd62b74412ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f28b4329d91c44b98ef80b977708d167"
I20260620 18:35:46.177462 26709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a2ea5623bed47b2837dcb3f4bda17b4" candidate_uuid: "41e3a33ef4884818ab9abd62b74412ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0201587fac944f7a411a55384847506"
I20260620 18:35:46.177557 26442 raft_consensus.cc:3037] T 3a2ea5623bed47b2837dcb3f4bda17b4 P f28b4329d91c44b98ef80b977708d167 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:46.177681 26709 raft_consensus.cc:3037] T 3a2ea5623bed47b2837dcb3f4bda17b4 P e0201587fac944f7a411a55384847506 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:35:46.179752 26442 raft_consensus.cc:2454] T 3a2ea5623bed47b2837dcb3f4bda17b4 P f28b4329d91c44b98ef80b977708d167 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41e3a33ef4884818ab9abd62b74412ac in term 1.
I20260620 18:35:46.179931 26709 raft_consensus.cc:2454] T 3a2ea5623bed47b2837dcb3f4bda17b4 P e0201587fac944f7a411a55384847506 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41e3a33ef4884818ab9abd62b74412ac in term 1.
I20260620 18:35:46.180138 26509 leader_election.cc:302] T 3a2ea5623bed47b2837dcb3f4bda17b4 P 41e3a33ef4884818ab9abd62b74412ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41e3a33ef4884818ab9abd62b74412ac, f28b4329d91c44b98ef80b977708d167; no voters:
I20260620 18:35:46.180420 26778 raft_consensus.cc:2781] T 3a2ea5623bed47b2837dcb3f4bda17b4 P 41e3a33ef4884818ab9abd62b74412ac [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:35:46.180886 26778 raft_consensus.cc:686] T 3a2ea5623bed47b2837dcb3f4bda17b4 P 41e3a33ef4884818ab9abd62b74412ac [term 1 LEADER]: Becoming Leader. State: Replica: 41e3a33ef4884818ab9abd62b74412ac, State: Running, Role: LEADER
I20260620 18:35:46.181353 26778 consensus_queue.cc:237] T 3a2ea5623bed47b2837dcb3f4bda17b4 P 41e3a33ef4884818ab9abd62b74412ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41e3a33ef4884818ab9abd62b74412ac" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40545 } } peers { permanent_uuid: "f28b4329d91c44b98ef80b977708d167" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40323 } } peers { permanent_uuid: "e0201587fac944f7a411a55384847506" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 33301 } }
I20260620 18:35:46.186216 26298 catalog_manager.cc:5495] T 3a2ea5623bed47b2837dcb3f4bda17b4 P 41e3a33ef4884818ab9abd62b74412ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 41e3a33ef4884818ab9abd62b74412ac (127.10.131.2). New cstate: current_term: 1 leader_uuid: "41e3a33ef4884818ab9abd62b74412ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41e3a33ef4884818ab9abd62b74412ac" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f28b4329d91c44b98ef80b977708d167" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0201587fac944f7a411a55384847506" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 33301 } health_report { overall_health: UNKNOWN } } }
W20260620 18:35:46.206135 26488 tablet.cc:2369] T 3a2ea5623bed47b2837dcb3f4bda17b4 P f28b4329d91c44b98ef80b977708d167: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:35:46.210498 26767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:46.204775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34204 (local address 127.10.131.2:40545)
0620 18:35:46.205029 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:46.205043 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:46.205061 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:35:46.205145 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:46.205148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:46.205224 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:46.205346 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:35:46.205729 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.206838 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.208405 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.208800 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.208955 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:46.209391 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:35:46.209893 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:35:46.209999 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:46.210130 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 18:35:46.210195 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
W20260620 18:35:46.212070 26622 tablet.cc:2369] T 3a2ea5623bed47b2837dcb3f4bda17b4 P 41e3a33ef4884818ab9abd62b74412ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:35:46.218984 26442 raft_consensus.cc:1261] T 3a2ea5623bed47b2837dcb3f4bda17b4 P f28b4329d91c44b98ef80b977708d167 [term 1 FOLLOWER]: Refusing update from remote peer 41e3a33ef4884818ab9abd62b74412ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:35:46.218982 26709 raft_consensus.cc:1261] T 3a2ea5623bed47b2837dcb3f4bda17b4 P e0201587fac944f7a411a55384847506 [term 1 FOLLOWER]: Refusing update from remote peer 41e3a33ef4884818ab9abd62b74412ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:35:46.220016 26783 consensus_queue.cc:1035] T 3a2ea5623bed47b2837dcb3f4bda17b4 P 41e3a33ef4884818ab9abd62b74412ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28b4329d91c44b98ef80b977708d167" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:35:46.220359 26778 consensus_queue.cc:1035] T 3a2ea5623bed47b2837dcb3f4bda17b4 P 41e3a33ef4884818ab9abd62b74412ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0201587fac944f7a411a55384847506" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 33301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 18:35:46.221553 26755 tablet.cc:2369] T 3a2ea5623bed47b2837dcb3f4bda17b4 P e0201587fac944f7a411a55384847506: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:35:46.233723 26788 mvcc.cc:205] Tried to move back new op lower bound from 7298992317303353344 to 7298992317163323392. Current Snapshot: MvccSnapshot[applied={T|T < 7298992317303353344}]
I20260620 18:35:46.234848 26786 mvcc.cc:205] Tried to move back new op lower bound from 7298992317303353344 to 7298992317163323392. Current Snapshot: MvccSnapshot[applied={T|T < 7298992317303353344}]
I20260620 18:35:46.235237 26787 mvcc.cc:205] Tried to move back new op lower bound from 7298992317303353344 to 7298992317163323392. Current Snapshot: MvccSnapshot[applied={T|T < 7298992317303353344}]
I20260620 18:35:46.248132 26298 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:35:46.248459 26298 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:35:46.252209 26298 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c: Sending DeleteTablet for 3 replicas of tablet 3a2ea5623bed47b2837dcb3f4bda17b4
I20260620 18:35:46.253454 26556 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a2ea5623bed47b2837dcb3f4bda17b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:35:46 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:34198
I20260620 18:35:46.253592 26689 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a2ea5623bed47b2837dcb3f4bda17b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:35:46 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:55994
I20260620 18:35:46.253706 26422 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a2ea5623bed47b2837dcb3f4bda17b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:35:46 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:59092
I20260620 18:35:46.253902 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 26359
I20260620 18:35:46.265928 26797 tablet_replica.cc:330] T 3a2ea5623bed47b2837dcb3f4bda17b4 P f28b4329d91c44b98ef80b977708d167: stopping tablet replica
I20260620 18:35:46.266543 26796 tablet_replica.cc:330] T 3a2ea5623bed47b2837dcb3f4bda17b4 P e0201587fac944f7a411a55384847506: stopping tablet replica
I20260620 18:35:46.267112 26796 raft_consensus.cc:2229] T 3a2ea5623bed47b2837dcb3f4bda17b4 P e0201587fac944f7a411a55384847506 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:35:46.267434 26795 tablet_replica.cc:330] T 3a2ea5623bed47b2837dcb3f4bda17b4 P 41e3a33ef4884818ab9abd62b74412ac: stopping tablet replica
I20260620 18:35:46.267656 26797 raft_consensus.cc:2229] T 3a2ea5623bed47b2837dcb3f4bda17b4 P f28b4329d91c44b98ef80b977708d167 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:35:46.267951 26795 raft_consensus.cc:2229] T 3a2ea5623bed47b2837dcb3f4bda17b4 P 41e3a33ef4884818ab9abd62b74412ac [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:35:46.268472 26796 raft_consensus.cc:2258] T 3a2ea5623bed47b2837dcb3f4bda17b4 P e0201587fac944f7a411a55384847506 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:35:46.268724 26797 raft_consensus.cc:2258] T 3a2ea5623bed47b2837dcb3f4bda17b4 P f28b4329d91c44b98ef80b977708d167 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:35:46.268760 26795 raft_consensus.cc:2258] T 3a2ea5623bed47b2837dcb3f4bda17b4 P 41e3a33ef4884818ab9abd62b74412ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:35:46.271091 26796 ts_tablet_manager.cc:1905] T 3a2ea5623bed47b2837dcb3f4bda17b4 P e0201587fac944f7a411a55384847506: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:35:46.271101 26795 ts_tablet_manager.cc:1905] T 3a2ea5623bed47b2837dcb3f4bda17b4 P 41e3a33ef4884818ab9abd62b74412ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:35:46.273173 26797 ts_tablet_manager.cc:1905] T 3a2ea5623bed47b2837dcb3f4bda17b4 P f28b4329d91c44b98ef80b977708d167: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 18:35:46 dist-test-slave-cbn3 krb5kdc[26253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.1@KRBTEST.COM
I20260620 18:35:46.280499 26797 ts_tablet_manager.cc:1918] T 3a2ea5623bed47b2837dcb3f4bda17b4 P f28b4329d91c44b98ef80b977708d167: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:35:46.280777 26797 log.cc:1196] T 3a2ea5623bed47b2837dcb3f4bda17b4 P f28b4329d91c44b98ef80b977708d167: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/wal/wals/3a2ea5623bed47b2837dcb3f4bda17b4
I20260620 18:35:46.281214 26796 ts_tablet_manager.cc:1918] T 3a2ea5623bed47b2837dcb3f4bda17b4 P e0201587fac944f7a411a55384847506: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:35:46.281394 26796 log.cc:1196] T 3a2ea5623bed47b2837dcb3f4bda17b4 P e0201587fac944f7a411a55384847506: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/wal/wals/3a2ea5623bed47b2837dcb3f4bda17b4
I20260620 18:35:46.281515 26797 ts_tablet_manager.cc:1939] T 3a2ea5623bed47b2837dcb3f4bda17b4 P f28b4329d91c44b98ef80b977708d167: Deleting consensus metadata
I20260620 18:35:46.281932 26796 ts_tablet_manager.cc:1939] T 3a2ea5623bed47b2837dcb3f4bda17b4 P e0201587fac944f7a411a55384847506: Deleting consensus metadata
I20260620 18:35:46.283516 26282 catalog_manager.cc:4841] TS f28b4329d91c44b98ef80b977708d167 (127.10.131.1:40323): tablet 3a2ea5623bed47b2837dcb3f4bda17b4 (table test-table [id=8d071f02d2614a808c4097d45b43496a]) successfully deleted
I20260620 18:35:46.283860 26282 catalog_manager.cc:4841] TS e0201587fac944f7a411a55384847506 (127.10.131.3:33301): tablet 3a2ea5623bed47b2837dcb3f4bda17b4 (table test-table [id=8d071f02d2614a808c4097d45b43496a]) successfully deleted
I20260620 18:35:46.284301 26795 ts_tablet_manager.cc:1918] T 3a2ea5623bed47b2837dcb3f4bda17b4 P 41e3a33ef4884818ab9abd62b74412ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:35:46.284487 26795 log.cc:1196] T 3a2ea5623bed47b2837dcb3f4bda17b4 P 41e3a33ef4884818ab9abd62b74412ac: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/wal/wals/3a2ea5623bed47b2837dcb3f4bda17b4
I20260620 18:35:46.285061 26795 ts_tablet_manager.cc:1939] T 3a2ea5623bed47b2837dcb3f4bda17b4 P 41e3a33ef4884818ab9abd62b74412ac: Deleting consensus metadata
I20260620 18:35:46.286715 26283 catalog_manager.cc:4841] TS 41e3a33ef4884818ab9abd62b74412ac (127.10.131.2:40545): tablet 3a2ea5623bed47b2837dcb3f4bda17b4 (table test-table [id=8d071f02d2614a808c4097d45b43496a]) successfully deleted
I20260620 18:35:46.291656 26769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:46.255387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59096 (local address 127.10.131.1:40323)
0620 18:35:46.255631 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:46.255641 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:46.267253 (+ 11612us) server_negotiation.cc:407] Connection header received
0620 18:35:46.267708 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:46.267715 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:46.267816 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:46.267921 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:46.270456 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.271488 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.273638 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.274353 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.274463 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:46.285002 (+ 10539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:46.285024 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:46.285041 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:46.285089 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:46.288896 (+ 3807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:46.289590 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:46.289600 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:46.289603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:46.289754 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:46.290096 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:46.290101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:46.290105 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:46.290547 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:46.290869 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:46.291104 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 18:35:46.291310 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 18:35:46.292351 26482 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:46.381065 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 26359
I20260620 18:35:46.399495 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 26493
Jun 20 18:35:46 dist-test-slave-cbn3 krb5kdc[26253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.2@KRBTEST.COM
I20260620 18:35:46.418422 26767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:46.400345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34208 (local address 127.10.131.2:40545)
0620 18:35:46.400589 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:46.400596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:46.400618 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:35:46.400683 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:46.400688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:46.400768 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:46.400917 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:46.402264 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.403575 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.405393 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.406124 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.406258 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:46.411310 (+ 5052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:46.411328 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:46.411348 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:46.411394 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:46.415514 (+ 4120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:46.416187 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:46.416197 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:46.416201 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:46.416383 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:46.416747 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:46.416752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:46.416756 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:46.417258 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:46.417561 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:46.417765 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 18:35:46.418010 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260620 18:35:46.419215 26616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:46.531309 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 26493
I20260620 18:35:46.549769 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 26626
Jun 20 18:35:46 dist-test-slave-cbn3 krb5kdc[26253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.3@KRBTEST.COM
I20260620 18:35:46.565570 26771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:35:46.550519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56002 (local address 127.10.131.3:33301)
0620 18:35:46.550703 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 18:35:46.550711 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:35:46.550731 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:35:46.550947 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:35:46.550952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:35:46.551032 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:35:46.551134 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:35:46.552640 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.553414 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.554682 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:35:46.555082 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:35:46.555389 (+ 307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:35:46.560036 (+ 4647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:35:46.560050 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:35:46.560066 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:35:46.560100 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:35:46.562675 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:46.563355 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:46.563368 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:46.563371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:46.563510 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:35:46.563874 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:35:46.563880 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:35:46.563884 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:35:46.564265 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:35:46.564552 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0620 18:35:46.565068 (+ 516us) server_negotiation.cc:299] Negotiation successful
0620 18:35:46.565295 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:35:46.566175 26749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:46.653553 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 26626
I20260620 18:35:46.670027 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 26266
I20260620 18:35:46.671183 26328 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:35:46.779482 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 26266
I20260620 18:36:06.800261 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:38121
--webserver_interface=127.10.131.62
--webserver_port=33489
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:37841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:38121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:07.132128 26803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:07.132407 26803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:07.132535 26803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:07.139859 26803 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 18:36:07.139966 26803 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 18:36:07.140044 26803 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 18:36:07.140118 26803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:36:07.140224 26803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:07.140343 26803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:07.140432 26803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:36:07.140513 26803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:36:07.150769 26803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:37841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:38121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:38121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=33489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:36:07.152274 26803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:07.153978 26803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:07.181296 26812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:07.181296 26809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:07.181340 26810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:07.186616 26803 server_base.cc:1034] running on GCE node
I20260620 18:36:07.187531 26803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:07.189232 26803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:07.190492 26803 hybrid_clock.cc:648] HybridClock initialized: now 1781980567190415 us; error 47 us; skew 500 ppm
I20260620 18:36:07.190979 26803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:07 dist-test-slave-cbn3 krb5kdc[26253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980567, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:07.199904 26803 init.cc:377] Logged in from keytab as kudu/127.10.131.62@KRBTEST.COM (short username kudu)
I20260620 18:36:07.201817 26803 webserver.cc:460] Webserver started at http://127.10.131.62:33489/ using document root <none> and password file <none>
I20260620 18:36:07.202656 26803 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:07.202792 26803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:07.208673 26803 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 18:36:07.211737 26819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:07.213249 26803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:36:07.213459 26803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "219191504a5243ac8cc1fcb988930d8c"
format_stamp: "Formatted at 2026-06-20 18:35:44 on dist-test-slave-cbn3"
server_key: "741e9131d0eb1922876e15143dd5e7ef"
server_key_iv: "8e9e30de5f3617c4417b9744be3b87b2"
server_key_version: "encryptionkey@0"
I20260620 18:36:07.214658 26803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:07.225694 26803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:07.230067 26803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:07.230360 26803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:07.249446 26803 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:38121
I20260620 18:36:07.249476 26870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:38121 every 8 connection(s)
I20260620 18:36:07.250953 26803 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:36:07.256088 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 26803
I20260620 18:36:07.257121 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.1:40323
--local_ip_for_outbound_sockets=127.10.131.1
--tserver_master_addrs=127.10.131.62:38121
--webserver_port=37115
--webserver_interface=127.10.131.1
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:37841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:36:07.259064 26871 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:36:07.264396 26871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c: Bootstrap starting.
I20260620 18:36:07.286365 26871 log.cc:824] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:07.300004 26871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:36:07.300664 26871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c: Bootstrap complete.
I20260620 18:36:07.311489 26871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219191504a5243ac8cc1fcb988930d8c" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38121 } }
I20260620 18:36:07.312493 26871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 219191504a5243ac8cc1fcb988930d8c, State: Initialized, Role: FOLLOWER
I20260620 18:36:07.313323 26871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219191504a5243ac8cc1fcb988930d8c" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38121 } }
I20260620 18:36:07.313601 26871 raft_consensus.cc:388] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:36:07.313748 26871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:36:07.313964 26871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:36:07.318550 26871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219191504a5243ac8cc1fcb988930d8c" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38121 } }
I20260620 18:36:07.319231 26871 leader_election.cc:302] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 219191504a5243ac8cc1fcb988930d8c; no voters:
I20260620 18:36:07.319985 26871 leader_election.cc:288] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 18:36:07.320137 26875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [term 2 FOLLOWER]: Leader election won for term 2
I20260620 18:36:07.321287 26875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [term 2 LEADER]: Becoming Leader. State: Replica: 219191504a5243ac8cc1fcb988930d8c, State: Running, Role: LEADER
I20260620 18:36:07.321858 26875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219191504a5243ac8cc1fcb988930d8c" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38121 } }
I20260620 18:36:07.322458 26871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:36:07.324525 26876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "219191504a5243ac8cc1fcb988930d8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219191504a5243ac8cc1fcb988930d8c" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38121 } } }
I20260620 18:36:07.325030 26876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:07.329001 26877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 219191504a5243ac8cc1fcb988930d8c. Latest consensus state: current_term: 2 leader_uuid: "219191504a5243ac8cc1fcb988930d8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219191504a5243ac8cc1fcb988930d8c" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38121 } } }
I20260620 18:36:07.329429 26877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:07.331131 26881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:36:07.339926 26881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:36:07.341382 26881 catalog_manager.cc:1259] Loaded cluster ID: b6c54873d26c4dc996c1efb81e4b15b9
I20260620 18:36:07.341507 26881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:36:07.347129 26881 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:36:07.348615 26881 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c: Loaded TSK: 0
I20260620 18:36:07.378628 26881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c: Generated new TSK 1
I20260620 18:36:07.378988 26881 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 18:36:07.609319 26873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:07.609709 26873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:07.609889 26873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:07.619453 26873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:07.619553 26873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:07.619724 26873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.1
I20260620 18:36:07.629771 26873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:37841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.1:40323
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.131.1
--webserver_port=37115
--tserver_master_addrs=127.10.131.62:38121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.1
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:36:07.631067 26873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:07.632375 26873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:07.647430 26899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:07.648221 26898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:07.648787 26901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:07.650147 26873 server_base.cc:1034] running on GCE node
I20260620 18:36:07.650856 26873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:07.651873 26873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:07.653091 26873 hybrid_clock.cc:648] HybridClock initialized: now 1781980567653044 us; error 43 us; skew 500 ppm
I20260620 18:36:07.653461 26873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:07 dist-test-slave-cbn3 krb5kdc[26253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980567, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:07.656817 26873 init.cc:377] Logged in from keytab as kudu/127.10.131.1@KRBTEST.COM (short username kudu)
I20260620 18:36:07.658643 26873 webserver.cc:460] Webserver started at http://127.10.131.1:37115/ using document root <none> and password file <none>
I20260620 18:36:07.659380 26873 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:07.659538 26873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:07.665190 26873 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 18:36:07.668311 26908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:07.669872 26873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:36:07.670045 26873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/wal
uuid: "f28b4329d91c44b98ef80b977708d167"
format_stamp: "Formatted at 2026-06-20 18:35:44 on dist-test-slave-cbn3"
server_key: "5b37b6e9eb0024f30f53b378fd98782f"
server_key_iv: "9c0378c9b550135a00bc9d1cdb6ee6dd"
server_key_version: "encryptionkey@0"
I20260620 18:36:07.670995 26873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:07.696120 26873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:07.700541 26873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:07.700825 26873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:07.701996 26873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:07.703547 26873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:07.703667 26873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:07.703794 26873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:07.703923 26873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:07.745059 26873 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.1:40323
I20260620 18:36:07.745088 27020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.1:40323 every 8 connection(s)
I20260620 18:36:07.746505 26873 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
I20260620 18:36:07.755574 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 26873
I20260620 18:36:07.756275 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.2:40545
--local_ip_for_outbound_sockets=127.10.131.2
--tserver_master_addrs=127.10.131.62:38121
--webserver_port=33655
--webserver_interface=127.10.131.2
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:37841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:36:07 dist-test-slave-cbn3 krb5kdc[26253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980567, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:36:07.771544 27024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:07.749655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.1:49637)
0620 18:36:07.750538 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:07.750583 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:07.752344 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:07.753633 (+ 1289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:07.753640 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:07.754159 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:07.754676 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:07.754693 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:07.756289 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:07.756297 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:07.758509 (+ 2212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:07.758526 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:07.759503 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:07.759512 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:07.759819 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:07.760543 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:07.760576 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:07.763508 (+ 2932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:07.767745 (+ 4237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:07.767751 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:07.767762 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:07.768152 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:07.768637 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:07.768640 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:07.768642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:07.768832 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:07.769777 (+ 945us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:07.769783 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:07.770050 (+ 267us) client_negotiation.cc:769] Sending connection context
0620 18:36:07.770267 (+ 217us) client_negotiation.cc:240] Negotiation successful
0620 18:36:07.770613 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":366,"threads_started":1}
I20260620 18:36:07.772135 27023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:07.749378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:49637 (local address 127.10.131.62:38121)
0620 18:36:07.749941 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:07.749953 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:07.750704 (+ 751us) server_negotiation.cc:407] Connection header received
0620 18:36:07.752818 (+ 2114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:07.752830 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:07.753204 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:07.753508 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:07.754857 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:07.756118 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:07.758647 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:07.759276 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:07.759405 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:07.763790 (+ 4385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:07.763801 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:07.763820 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:07.763876 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:07.767585 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:07.768307 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:07.768318 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:07.768328 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:07.768531 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:07.769011 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:07.769018 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:07.769025 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:07.769652 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:07.769836 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:07.770540 (+ 704us) server_negotiation.cc:299] Negotiation successful
0620 18:36:07.770953 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":236,"threads_started":1}
I20260620 18:36:07.774761 27021 heartbeater.cc:347] Connected to a master server at 127.10.131.62:38121
I20260620 18:36:07.775081 27021 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:07.776007 27021 heartbeater.cc:511] Master 127.10.131.62:38121 requested a full tablet report, sending...
I20260620 18:36:07.778329 26836 ts_manager.cc:194] Registered new tserver with Master: f28b4329d91c44b98ef80b977708d167 (127.10.131.1:40323)
I20260620 18:36:07.779835 26836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.1@KRBTEST.COM'} at 127.10.131.1:49637
W20260620 18:36:08.083118 27025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:08.083443 27025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:08.083616 27025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:08.090920 27025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:08.091079 27025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:08.091280 27025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.2
I20260620 18:36:08.101503 27025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:37841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.2:40545
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.131.2
--webserver_port=33655
--tserver_master_addrs=127.10.131.62:38121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.2
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:36:08.102900 27025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:08.104256 27025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:08.117518 27030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:08.117532 27031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:08.117666 27033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:08.118263 27025 server_base.cc:1034] running on GCE node
I20260620 18:36:08.118884 27025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:08.119791 27025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:08.120939 27025 hybrid_clock.cc:648] HybridClock initialized: now 1781980568120906 us; error 30 us; skew 500 ppm
I20260620 18:36:08.121201 27025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:08 dist-test-slave-cbn3 krb5kdc[26253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980568, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:08.124344 27025 init.cc:377] Logged in from keytab as kudu/127.10.131.2@KRBTEST.COM (short username kudu)
I20260620 18:36:08.125800 27025 webserver.cc:460] Webserver started at http://127.10.131.2:33655/ using document root <none> and password file <none>
I20260620 18:36:08.126459 27025 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:08.126607 27025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:08.131834 27025 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260620 18:36:08.134526 27040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:08.135836 27025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 18:36:08.136011 27025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/wal
uuid: "41e3a33ef4884818ab9abd62b74412ac"
format_stamp: "Formatted at 2026-06-20 18:35:45 on dist-test-slave-cbn3"
server_key: "5b944fe391bc39257d5882455852ec03"
server_key_iv: "6268bea237db663cdf38df60554e3ab5"
server_key_version: "encryptionkey@0"
I20260620 18:36:08.137009 27025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:08.150354 27025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:08.153973 27025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:08.154258 27025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:08.155153 27025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:08.156445 27025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:08.156569 27025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:08.156699 27025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:08.156790 27025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:08.190495 27025 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.2:40545
I20260620 18:36:08.190575 27152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.2:40545 every 8 connection(s)
I20260620 18:36:08.191862 27025 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
I20260620 18:36:08.193341 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 27025
I20260620 18:36:08.193944 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.3:33301
--local_ip_for_outbound_sockets=127.10.131.3
--tserver_master_addrs=127.10.131.62:38121
--webserver_port=44637
--webserver_interface=127.10.131.3
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:37841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:36:08 dist-test-slave-cbn3 krb5kdc[26253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980568, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:36:08.214299 27023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:08.195299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:35385 (local address 127.10.131.62:38121)
0620 18:36:08.195485 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:08.195491 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:08.195803 (+ 312us) server_negotiation.cc:407] Connection header received
0620 18:36:08.198654 (+ 2851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:08.198658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:08.198746 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:08.198838 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:08.201013 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:08.201790 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:08.203906 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:08.204304 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:08.204397 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:08.208475 (+ 4078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:08.208479 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:08.208483 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:08.208516 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:08.211155 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:08.211870 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:08.211874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:08.211876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:08.211992 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:08.212414 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:08.212419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:08.212422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:08.212816 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:08.212995 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:08.213702 (+ 707us) server_negotiation.cc:299] Negotiation successful
0620 18:36:08.213944 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:36:08.215305 27155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:08.194819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.2:35385)
0620 18:36:08.195615 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:08.195662 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:08.198315 (+ 2653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:08.199131 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:08.199146 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:08.199943 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:08.200790 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:08.200815 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:08.201966 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:08.201974 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:08.203769 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:08.203786 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:08.204520 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:08.204529 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:08.204872 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:08.205599 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:08.205632 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:08.208244 (+ 2612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:08.211302 (+ 3058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:08.211311 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:08.211328 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:08.211738 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:08.212097 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:08.212102 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:08.212105 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:08.212293 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:08.212962 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:08.212970 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:08.213329 (+ 359us) client_negotiation.cc:769] Sending connection context
0620 18:36:08.213594 (+ 265us) client_negotiation.cc:240] Negotiation successful
0620 18:36:08.214129 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":299,"threads_started":1}
I20260620 18:36:08.216773 27153 heartbeater.cc:347] Connected to a master server at 127.10.131.62:38121
I20260620 18:36:08.217098 27153 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:08.217862 27153 heartbeater.cc:511] Master 127.10.131.62:38121 requested a full tablet report, sending...
I20260620 18:36:08.219259 26836 ts_manager.cc:194] Registered new tserver with Master: 41e3a33ef4884818ab9abd62b74412ac (127.10.131.2:40545)
I20260620 18:36:08.220144 26836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.2@KRBTEST.COM'} at 127.10.131.2:35385
W20260620 18:36:08.518829 27156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:08.519129 27156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:08.519290 27156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:08.526876 27156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:08.527029 27156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:08.527210 27156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.3
I20260620 18:36:08.537073 27156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:37841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.3:33301
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.131.3
--webserver_port=44637
--tserver_master_addrs=127.10.131.62:38121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.3
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:36:08.538357 27156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:08.539893 27156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:08.553447 27161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:08.553399 27162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:08.553368 27164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:08.558596 27156 server_base.cc:1034] running on GCE node
I20260620 18:36:08.559226 27156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:08.560163 27156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:08.561422 27156 hybrid_clock.cc:648] HybridClock initialized: now 1781980568561365 us; error 44 us; skew 500 ppm
I20260620 18:36:08.561748 27156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:08 dist-test-slave-cbn3 krb5kdc[26253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980568, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:08.565007 27156 init.cc:377] Logged in from keytab as kudu/127.10.131.3@KRBTEST.COM (short username kudu)
I20260620 18:36:08.566387 27156 webserver.cc:460] Webserver started at http://127.10.131.3:44637/ using document root <none> and password file <none>
I20260620 18:36:08.567019 27156 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:08.567124 27156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:08.572260 27156 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:36:08.574968 27171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:08.576272 27156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:36:08.576427 27156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/wal
uuid: "e0201587fac944f7a411a55384847506"
format_stamp: "Formatted at 2026-06-20 18:35:45 on dist-test-slave-cbn3"
server_key: "0c42b0ce96c7bc82cae239b8700e7d1a"
server_key_iv: "4a2bcd7a747c20d3e1a8653578c680d5"
server_key_version: "encryptionkey@0"
I20260620 18:36:08.577613 27156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:08.605412 27156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:08.609551 27156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:08.609774 27156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:08.610677 27156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:08.611908 27156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:08.611989 27156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:08.612058 27156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:08.612110 27156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:08.646075 27156 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.3:33301
I20260620 18:36:08.646159 27283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.3:33301 every 8 connection(s)
I20260620 18:36:08.647451 27156 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
I20260620 18:36:08.649967 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 27156
Jun 20 18:36:08 dist-test-slave-cbn3 krb5kdc[26253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980568, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:36:08.667593 27286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:08.650983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47492 (local address 127.10.131.62:38121)
0620 18:36:08.651502 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:08.651510 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:08.653443 (+ 1933us) server_negotiation.cc:407] Connection header received
0620 18:36:08.653691 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:08.653700 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:08.653762 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:08.653875 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:08.655375 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:08.656128 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:08.657644 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:08.658094 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:08.658224 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:08.661424 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:08.661436 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:08.661440 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:08.661476 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:08.664654 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:08.665457 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:08.665462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:08.665464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:08.665608 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:08.666020 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:08.666030 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:08.666032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:08.666315 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:08.666605 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:08.666901 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 18:36:08.667133 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":257,"threads_started":1}
I20260620 18:36:08.669622 27023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:08.650381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:43551 (local address 127.10.131.62:38121)
0620 18:36:08.650593 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:08.650600 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:08.653612 (+ 3012us) server_negotiation.cc:407] Connection header received
0620 18:36:08.655429 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:08.655434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:08.655511 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:08.655634 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:08.657134 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:08.657827 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:08.659395 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:08.659808 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:08.659938 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:08.663236 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:08.663241 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:08.663244 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:08.663283 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:08.666153 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:08.666877 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:08.666886 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:08.666888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:08.666995 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:08.667516 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:08.667523 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:08.667528 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:08.667880 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:08.668034 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:08.669000 (+ 966us) server_negotiation.cc:299] Negotiation successful
0620 18:36:08.669261 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:36:08.670769 27288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:08.650983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.3:43551)
0620 18:36:08.653454 (+ 2471us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:08.653505 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:08.655171 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:08.655867 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:08.655874 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:08.656367 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:08.656977 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:08.656996 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:08.657991 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:08.657996 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:08.659229 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:08.659241 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:08.660026 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:08.660033 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:08.660258 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:08.660936 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:08.660955 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:08.663031 (+ 2076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:08.666298 (+ 3267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:08.666307 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:08.666324 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:08.666748 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:08.667103 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:08.667111 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:08.667115 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:08.667324 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:08.668138 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:08.668152 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:08.668526 (+ 374us) client_negotiation.cc:769] Sending connection context
0620 18:36:08.668872 (+ 346us) client_negotiation.cc:240] Negotiation successful
0620 18:36:08.669437 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2181,"thread_start_us":1938,"threads_started":1}
I20260620 18:36:08.672027 27284 heartbeater.cc:347] Connected to a master server at 127.10.131.62:38121
I20260620 18:36:08.672325 27284 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:08.673147 27284 heartbeater.cc:511] Master 127.10.131.62:38121 requested a full tablet report, sending...
I20260620 18:36:08.674496 26836 ts_manager.cc:194] Registered new tserver with Master: e0201587fac944f7a411a55384847506 (127.10.131.3:33301)
I20260620 18:36:08.675263 26836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.3@KRBTEST.COM'} at 127.10.131.3:43551
I20260620 18:36:08.682827 10764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:36:08.689522 27023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:08.683750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47502 (local address 127.10.131.62:38121)
0620 18:36:08.683952 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:08.683958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:08.684357 (+ 399us) server_negotiation.cc:407] Connection header received
0620 18:36:08.684597 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:08.684601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:08.684723 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:08.684896 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:36:08.685353 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:08.686091 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:08.687616 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:08.688002 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:08.688135 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:08.688650 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:36:08.688839 (+ 189us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 18:36:08.689258 (+ 419us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47502: token expired
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 18:36:08.689662 27023 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47502: token expired
W20260620 18:36:08.690119 26759 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.10.131.62:38121)
I20260620 18:36:08.702940 27023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:08.690584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47506 (local address 127.10.131.62:38121)
0620 18:36:08.690736 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:08.690740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:08.690755 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:36:08.690875 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:08.690879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:08.690944 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:08.691020 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:08.692343 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:08.693115 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:08.694465 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:08.694861 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:08.694951 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:08.697575 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:08.697580 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:08.697583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:08.697622 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:08.699997 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:08.700707 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:08.700712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:08.700714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:08.700909 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:08.701283 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:08.701287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:08.701289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:08.701737 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:08.702015 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:08.702279 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 18:36:08.702591 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:36:08.706280 26836 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47506:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:36:08.713420 26836 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:36:08.735517 27296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:08.725273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:40545 (local address 127.0.0.1:50466)
0620 18:36:08.726783 (+ 1510us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:08.726799 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:08.726916 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:08.727611 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:08.727618 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:08.727646 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:08.728020 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:08.728033 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:08.730055 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:08.730063 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:08.732579 (+ 2516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:08.732591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:08.734472 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:08.734479 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:08.734679 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:08.734714 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 18:36:08.734852 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 18:36:08.735042 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1397,"mutex_wait_us":158,"thread_start_us":1611,"threads_started":1}
I20260620 18:36:08.735710 27297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:08.725747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50466 (local address 127.10.131.2:40545)
0620 18:36:08.726280 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:08.726286 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:08.727054 (+ 768us) server_negotiation.cc:407] Connection header received
0620 18:36:08.727154 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:08.727163 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:08.727379 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:08.727611 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:08.728233 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:08.729895 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:08.732729 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:08.734238 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:08.734390 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:08.734593 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:08.734891 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 18:36:08.735364 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":456,"threads_started":1}
I20260620 18:36:08.736150 27294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:08.724645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:33301 (local address 127.0.0.1:54970)
0620 18:36:08.725964 (+ 1319us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:08.726006 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:08.726217 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:08.727976 (+ 1759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:08.727984 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:08.728020 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:08.728433 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:08.728443 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:08.730268 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:08.730285 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:08.733019 (+ 2734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:08.733033 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:08.735244 (+ 2211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:08.735250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:08.735431 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:08.735473 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 18:36:08.735586 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 18:36:08.735727 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"mutex_wait_us":3,"thread_start_us":561,"threads_started":1}
I20260620 18:36:08.736826 27298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:08.726785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:40323 (local address 127.0.0.1:33498)
0620 18:36:08.727370 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:08.727393 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:08.727535 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:08.729081 (+ 1546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:08.729086 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:08.729110 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:08.729424 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:08.729433 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:08.730935 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:08.730940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:08.733494 (+ 2554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:08.733507 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:08.735562 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:08.735569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:08.736158 (+ 589us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:08.736200 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 18:36:08.736389 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 18:36:08.736513 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":546,"threads_started":1}
I20260620 18:36:08.736918 27295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:08.724999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54970 (local address 127.10.131.3:33301)
0620 18:36:08.725962 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:08.725971 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:08.726133 (+ 162us) server_negotiation.cc:407] Connection header received
0620 18:36:08.726447 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:08.726457 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:08.726663 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:08.727950 (+ 1287us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:08.728604 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:08.730128 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:08.733321 (+ 3193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:08.734743 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:08.734868 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:08.735074 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:08.736390 (+ 1316us) server_negotiation.cc:299] Negotiation successful
0620 18:36:08.736544 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":804,"thread_start_us":669,"threads_started":1}
I20260620 18:36:08.737502 27299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:08.727017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33498 (local address 127.10.131.1:40323)
0620 18:36:08.728546 (+ 1529us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:08.728557 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:08.728596 (+ 39us) server_negotiation.cc:407] Connection header received
0620 18:36:08.728679 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:08.728686 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:08.728904 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:08.729069 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:08.729599 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:08.730797 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:08.733667 (+ 2870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:08.735078 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:08.735219 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:08.735398 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:08.737174 (+ 1776us) server_negotiation.cc:299] Negotiation successful
0620 18:36:08.737298 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1378,"thread_start_us":1286,"threads_started":1}
I20260620 18:36:08.739665 27088 tablet_service.cc:1467] Processing CreateTablet for tablet e4b94d4b8eea4de491878f088c00f451 (DEFAULT_TABLE table=test-table [id=2db015a3431c4e48b942ccc98eadcb84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:08.739966 27219 tablet_service.cc:1467] Processing CreateTablet for tablet e4b94d4b8eea4de491878f088c00f451 (DEFAULT_TABLE table=test-table [id=2db015a3431c4e48b942ccc98eadcb84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:08.741184 27088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4b94d4b8eea4de491878f088c00f451. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:08.741307 27219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4b94d4b8eea4de491878f088c00f451. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:08.740906 26956 tablet_service.cc:1467] Processing CreateTablet for tablet e4b94d4b8eea4de491878f088c00f451 (DEFAULT_TABLE table=test-table [id=2db015a3431c4e48b942ccc98eadcb84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:08.742137 26956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4b94d4b8eea4de491878f088c00f451. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:08.757599 27301 tablet_bootstrap.cc:492] T e4b94d4b8eea4de491878f088c00f451 P 41e3a33ef4884818ab9abd62b74412ac: Bootstrap starting.
I20260620 18:36:08.760653 27301 tablet_bootstrap.cc:654] T e4b94d4b8eea4de491878f088c00f451 P 41e3a33ef4884818ab9abd62b74412ac: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:08.761591 27300 tablet_bootstrap.cc:492] T e4b94d4b8eea4de491878f088c00f451 P e0201587fac944f7a411a55384847506: Bootstrap starting.
I20260620 18:36:08.762389 27301 log.cc:824] T e4b94d4b8eea4de491878f088c00f451 P 41e3a33ef4884818ab9abd62b74412ac: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:08.765173 27303 tablet_bootstrap.cc:492] T e4b94d4b8eea4de491878f088c00f451 P f28b4329d91c44b98ef80b977708d167: Bootstrap starting.
I20260620 18:36:08.765328 27300 tablet_bootstrap.cc:654] T e4b94d4b8eea4de491878f088c00f451 P e0201587fac944f7a411a55384847506: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:08.766396 27301 tablet_bootstrap.cc:492] T e4b94d4b8eea4de491878f088c00f451 P 41e3a33ef4884818ab9abd62b74412ac: No bootstrap required, opened a new log
I20260620 18:36:08.766614 27300 log.cc:824] T e4b94d4b8eea4de491878f088c00f451 P e0201587fac944f7a411a55384847506: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:08.766681 27301 ts_tablet_manager.cc:1397] T e4b94d4b8eea4de491878f088c00f451 P 41e3a33ef4884818ab9abd62b74412ac: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 18:36:08.768640 27303 tablet_bootstrap.cc:654] T e4b94d4b8eea4de491878f088c00f451 P f28b4329d91c44b98ef80b977708d167: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:08.769766 27300 tablet_bootstrap.cc:492] T e4b94d4b8eea4de491878f088c00f451 P e0201587fac944f7a411a55384847506: No bootstrap required, opened a new log
I20260620 18:36:08.769874 27303 log.cc:824] T e4b94d4b8eea4de491878f088c00f451 P f28b4329d91c44b98ef80b977708d167: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:08.770031 27300 ts_tablet_manager.cc:1397] T e4b94d4b8eea4de491878f088c00f451 P e0201587fac944f7a411a55384847506: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 18:36:08.773526 27303 tablet_bootstrap.cc:492] T e4b94d4b8eea4de491878f088c00f451 P f28b4329d91c44b98ef80b977708d167: No bootstrap required, opened a new log
I20260620 18:36:08.773788 27303 ts_tablet_manager.cc:1397] T e4b94d4b8eea4de491878f088c00f451 P f28b4329d91c44b98ef80b977708d167: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260620 18:36:08.775208 27301 raft_consensus.cc:348] T e4b94d4b8eea4de491878f088c00f451 P 41e3a33ef4884818ab9abd62b74412ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0201587fac944f7a411a55384847506" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 33301 } } peers { permanent_uuid: "f28b4329d91c44b98ef80b977708d167" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40323 } } peers { permanent_uuid: "41e3a33ef4884818ab9abd62b74412ac" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40545 } }
I20260620 18:36:08.775573 27301 raft_consensus.cc:374] T e4b94d4b8eea4de491878f088c00f451 P 41e3a33ef4884818ab9abd62b74412ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:08.775672 27301 raft_consensus.cc:729] T e4b94d4b8eea4de491878f088c00f451 P 41e3a33ef4884818ab9abd62b74412ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41e3a33ef4884818ab9abd62b74412ac, State: Initialized, Role: FOLLOWER
I20260620 18:36:08.776242 27301 consensus_queue.cc:260] T e4b94d4b8eea4de491878f088c00f451 P 41e3a33ef4884818ab9abd62b74412ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0201587fac944f7a411a55384847506" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 33301 } } peers { permanent_uuid: "f28b4329d91c44b98ef80b977708d167" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40323 } } peers { permanent_uuid: "41e3a33ef4884818ab9abd62b74412ac" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40545 } }
I20260620 18:36:08.777626 27153 heartbeater.cc:503] Master 127.10.131.62:38121 was elected leader, sending a full tablet report...
I20260620 18:36:08.778208 27301 ts_tablet_manager.cc:1428] T e4b94d4b8eea4de491878f088c00f451 P 41e3a33ef4884818ab9abd62b74412ac: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260620 18:36:08.780525 27300 raft_consensus.cc:348] T e4b94d4b8eea4de491878f088c00f451 P e0201587fac944f7a411a55384847506 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0201587fac944f7a411a55384847506" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 33301 } } peers { permanent_uuid: "f28b4329d91c44b98ef80b977708d167" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40323 } } peers { permanent_uuid: "41e3a33ef4884818ab9abd62b74412ac" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40545 } }
I20260620 18:36:08.780992 27300 raft_consensus.cc:374] T e4b94d4b8eea4de491878f088c00f451 P e0201587fac944f7a411a55384847506 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:08.781093 27300 raft_consensus.cc:729] T e4b94d4b8eea4de491878f088c00f451 P e0201587fac944f7a411a55384847506 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0201587fac944f7a411a55384847506, State: Initialized, Role: FOLLOWER
I20260620 18:36:08.781706 27300 consensus_queue.cc:260] T e4b94d4b8eea4de491878f088c00f451 P e0201587fac944f7a411a55384847506 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0201587fac944f7a411a55384847506" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 33301 } } peers { permanent_uuid: "f28b4329d91c44b98ef80b977708d167" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40323 } } peers { permanent_uuid: "41e3a33ef4884818ab9abd62b74412ac" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40545 } }
I20260620 18:36:08.783617 27300 ts_tablet_manager.cc:1428] T e4b94d4b8eea4de491878f088c00f451 P e0201587fac944f7a411a55384847506: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260620 18:36:08.783730 27021 heartbeater.cc:503] Master 127.10.131.62:38121 was elected leader, sending a full tablet report...
I20260620 18:36:08.784025 27284 heartbeater.cc:503] Master 127.10.131.62:38121 was elected leader, sending a full tablet report...
I20260620 18:36:08.786628 27303 raft_consensus.cc:348] T e4b94d4b8eea4de491878f088c00f451 P f28b4329d91c44b98ef80b977708d167 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0201587fac944f7a411a55384847506" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 33301 } } peers { permanent_uuid: "f28b4329d91c44b98ef80b977708d167" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40323 } } peers { permanent_uuid: "41e3a33ef4884818ab9abd62b74412ac" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40545 } }
I20260620 18:36:08.786949 27303 raft_consensus.cc:374] T e4b94d4b8eea4de491878f088c00f451 P f28b4329d91c44b98ef80b977708d167 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:08.787038 27303 raft_consensus.cc:729] T e4b94d4b8eea4de491878f088c00f451 P f28b4329d91c44b98ef80b977708d167 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f28b4329d91c44b98ef80b977708d167, State: Initialized, Role: FOLLOWER
I20260620 18:36:08.787544 27303 consensus_queue.cc:260] T e4b94d4b8eea4de491878f088c00f451 P f28b4329d91c44b98ef80b977708d167 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0201587fac944f7a411a55384847506" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 33301 } } peers { permanent_uuid: "f28b4329d91c44b98ef80b977708d167" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40323 } } peers { permanent_uuid: "41e3a33ef4884818ab9abd62b74412ac" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40545 } }
I20260620 18:36:08.788892 27303 ts_tablet_manager.cc:1428] T e4b94d4b8eea4de491878f088c00f451 P f28b4329d91c44b98ef80b977708d167: Time spent starting tablet: real 0.015s user 0.012s sys 0.001s
W20260620 18:36:08.899165 27285 tablet.cc:2369] T e4b94d4b8eea4de491878f088c00f451 P e0201587fac944f7a411a55384847506: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:36:08.943910 27154 tablet.cc:2369] T e4b94d4b8eea4de491878f088c00f451 P 41e3a33ef4884818ab9abd62b74412ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:36:08.986162 27306 raft_consensus.cc:482] T e4b94d4b8eea4de491878f088c00f451 P 41e3a33ef4884818ab9abd62b74412ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:36:08.986478 27306 raft_consensus.cc:504] T e4b94d4b8eea4de491878f088c00f451 P 41e3a33ef4884818ab9abd62b74412ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0201587fac944f7a411a55384847506" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 33301 } } peers { permanent_uuid: "f28b4329d91c44b98ef80b977708d167" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40323 } } peers { permanent_uuid: "41e3a33ef4884818ab9abd62b74412ac" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40545 } }
I20260620 18:36:08.988106 27306 leader_election.cc:288] T e4b94d4b8eea4de491878f088c00f451 P 41e3a33ef4884818ab9abd62b74412ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0201587fac944f7a411a55384847506 (127.10.131.3:33301), f28b4329d91c44b98ef80b977708d167 (127.10.131.1:40323)
I20260620 18:36:08.994529 27295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:08.988575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:40447 (local address 127.10.131.3:33301)
0620 18:36:08.988760 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:08.988764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:08.989333 (+ 569us) server_negotiation.cc:407] Connection header received
0620 18:36:08.989571 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:08.989577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:08.989657 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:08.989747 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:08.990316 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:08.991022 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:08.992728 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:08.993597 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:08.993780 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:08.993855 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:08.994128 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 18:36:08.994265 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:36:08.994624 27309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:08.988597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:33301 (local address 127.10.131.2:40447)
0620 18:36:08.989217 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:08.989232 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:08.989407 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:08.989785 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:08.989790 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:08.989822 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:08.990173 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:08.990182 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:08.991167 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:08.991174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:08.992573 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:08.992585 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:08.993804 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:08.993811 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:08.993964 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:08.993987 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 18:36:08.994102 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 18:36:08.994230 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":334,"threads_started":1}
I20260620 18:36:08.995249 27299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:08.989486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:59147 (local address 127.10.131.1:40323)
0620 18:36:08.989697 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:08.989705 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:08.990141 (+ 436us) server_negotiation.cc:407] Connection header received
0620 18:36:08.990348 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:08.990351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:08.990416 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:08.990509 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:08.991026 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:08.991731 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:08.993459 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:08.994223 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:08.994417 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:08.994488 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:08.994834 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 18:36:08.994976 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:36:08.995241 27310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:08.989332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:40323 (local address 127.10.131.2:59147)
0620 18:36:08.990038 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:08.990052 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:08.990212 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:08.990520 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:08.990525 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:08.990546 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:08.990880 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:08.990895 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:08.991855 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:08.991865 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:08.993300 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:08.993311 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:08.994438 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:08.994444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:08.994647 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:08.994694 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 18:36:08.994796 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 18:36:08.994928 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":1,"thread_start_us":409,"threads_started":1}
I20260620 18:36:08.995776 27239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b94d4b8eea4de491878f088c00f451" candidate_uuid: "41e3a33ef4884818ab9abd62b74412ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0201587fac944f7a411a55384847506" is_pre_election: true
I20260620 18:36:08.996248 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b94d4b8eea4de491878f088c00f451" candidate_uuid: "41e3a33ef4884818ab9abd62b74412ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f28b4329d91c44b98ef80b977708d167" is_pre_election: true
I20260620 18:36:08.996369 27239 raft_consensus.cc:2454] T e4b94d4b8eea4de491878f088c00f451 P e0201587fac944f7a411a55384847506 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41e3a33ef4884818ab9abd62b74412ac in term 0.
I20260620 18:36:08.996780 26976 raft_consensus.cc:2454] T e4b94d4b8eea4de491878f088c00f451 P f28b4329d91c44b98ef80b977708d167 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41e3a33ef4884818ab9abd62b74412ac in term 0.
I20260620 18:36:08.997123 27041 leader_election.cc:302] T e4b94d4b8eea4de491878f088c00f451 P 41e3a33ef4884818ab9abd62b74412ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41e3a33ef4884818ab9abd62b74412ac, e0201587fac944f7a411a55384847506; no voters:
I20260620 18:36:08.997537 27306 raft_consensus.cc:2781] T e4b94d4b8eea4de491878f088c00f451 P 41e3a33ef4884818ab9abd62b74412ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:36:08.997701 27306 raft_consensus.cc:482] T e4b94d4b8eea4de491878f088c00f451 P 41e3a33ef4884818ab9abd62b74412ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:36:08.997809 27306 raft_consensus.cc:3037] T e4b94d4b8eea4de491878f088c00f451 P 41e3a33ef4884818ab9abd62b74412ac [term 0 FOLLOWER]: Advancing to term 1
W20260620 18:36:08.998741 27022 tablet.cc:2369] T e4b94d4b8eea4de491878f088c00f451 P f28b4329d91c44b98ef80b977708d167: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:36:09.000037 27306 raft_consensus.cc:504] T e4b94d4b8eea4de491878f088c00f451 P 41e3a33ef4884818ab9abd62b74412ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0201587fac944f7a411a55384847506" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 33301 } } peers { permanent_uuid: "f28b4329d91c44b98ef80b977708d167" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40323 } } peers { permanent_uuid: "41e3a33ef4884818ab9abd62b74412ac" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40545 } }
I20260620 18:36:09.000613 27306 leader_election.cc:288] T e4b94d4b8eea4de491878f088c00f451 P 41e3a33ef4884818ab9abd62b74412ac [CANDIDATE]: Term 1 election: Requested vote from peers e0201587fac944f7a411a55384847506 (127.10.131.3:33301), f28b4329d91c44b98ef80b977708d167 (127.10.131.1:40323)
I20260620 18:36:09.001145 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b94d4b8eea4de491878f088c00f451" candidate_uuid: "41e3a33ef4884818ab9abd62b74412ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f28b4329d91c44b98ef80b977708d167"
I20260620 18:36:09.001140 27239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b94d4b8eea4de491878f088c00f451" candidate_uuid: "41e3a33ef4884818ab9abd62b74412ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0201587fac944f7a411a55384847506"
I20260620 18:36:09.001363 26976 raft_consensus.cc:3037] T e4b94d4b8eea4de491878f088c00f451 P f28b4329d91c44b98ef80b977708d167 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:09.001363 27239 raft_consensus.cc:3037] T e4b94d4b8eea4de491878f088c00f451 P e0201587fac944f7a411a55384847506 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:09.004595 27239 raft_consensus.cc:2454] T e4b94d4b8eea4de491878f088c00f451 P e0201587fac944f7a411a55384847506 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41e3a33ef4884818ab9abd62b74412ac in term 1.
I20260620 18:36:09.004595 26976 raft_consensus.cc:2454] T e4b94d4b8eea4de491878f088c00f451 P f28b4329d91c44b98ef80b977708d167 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41e3a33ef4884818ab9abd62b74412ac in term 1.
I20260620 18:36:09.004999 27041 leader_election.cc:302] T e4b94d4b8eea4de491878f088c00f451 P 41e3a33ef4884818ab9abd62b74412ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41e3a33ef4884818ab9abd62b74412ac, f28b4329d91c44b98ef80b977708d167; no voters:
I20260620 18:36:09.005270 27306 raft_consensus.cc:2781] T e4b94d4b8eea4de491878f088c00f451 P 41e3a33ef4884818ab9abd62b74412ac [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:36:09.005709 27306 raft_consensus.cc:686] T e4b94d4b8eea4de491878f088c00f451 P 41e3a33ef4884818ab9abd62b74412ac [term 1 LEADER]: Becoming Leader. State: Replica: 41e3a33ef4884818ab9abd62b74412ac, State: Running, Role: LEADER
I20260620 18:36:09.006219 27306 consensus_queue.cc:237] T e4b94d4b8eea4de491878f088c00f451 P 41e3a33ef4884818ab9abd62b74412ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0201587fac944f7a411a55384847506" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 33301 } } peers { permanent_uuid: "f28b4329d91c44b98ef80b977708d167" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40323 } } peers { permanent_uuid: "41e3a33ef4884818ab9abd62b74412ac" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40545 } }
I20260620 18:36:09.010826 26835 catalog_manager.cc:5495] T e4b94d4b8eea4de491878f088c00f451 P 41e3a33ef4884818ab9abd62b74412ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 41e3a33ef4884818ab9abd62b74412ac (127.10.131.2). New cstate: current_term: 1 leader_uuid: "41e3a33ef4884818ab9abd62b74412ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0201587fac944f7a411a55384847506" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 33301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f28b4329d91c44b98ef80b977708d167" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41e3a33ef4884818ab9abd62b74412ac" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 40545 } health_report { overall_health: HEALTHY } } }
I20260620 18:36:09.039664 27297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:09.034230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50468 (local address 127.10.131.2:40545)
0620 18:36:09.034540 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:09.034549 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:09.034607 (+ 58us) server_negotiation.cc:407] Connection header received
0620 18:36:09.034855 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:09.034859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:09.034932 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:09.035082 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:36:09.035428 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:09.036157 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:09.037616 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:09.038001 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:09.038132 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:09.038622 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:36:09.039091 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:36:09.039244 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:09.039324 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 18:36:09.039401 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260620 18:36:09.047609 26976 raft_consensus.cc:1261] T e4b94d4b8eea4de491878f088c00f451 P f28b4329d91c44b98ef80b977708d167 [term 1 FOLLOWER]: Refusing update from remote peer 41e3a33ef4884818ab9abd62b74412ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:36:09.047610 27239 raft_consensus.cc:1261] T e4b94d4b8eea4de491878f088c00f451 P e0201587fac944f7a411a55384847506 [term 1 FOLLOWER]: Refusing update from remote peer 41e3a33ef4884818ab9abd62b74412ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:36:09.048548 27311 consensus_queue.cc:1035] T e4b94d4b8eea4de491878f088c00f451 P 41e3a33ef4884818ab9abd62b74412ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0201587fac944f7a411a55384847506" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 33301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:36:09.048933 27306 consensus_queue.cc:1035] T e4b94d4b8eea4de491878f088c00f451 P 41e3a33ef4884818ab9abd62b74412ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28b4329d91c44b98ef80b977708d167" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:36:09.061483 27313 mvcc.cc:205] Tried to move back new op lower bound from 7298992410809266176 to 7298992410653310976. Current Snapshot: MvccSnapshot[applied={T|T < 7298992410809266176}]
I20260620 18:36:09.062795 27315 mvcc.cc:205] Tried to move back new op lower bound from 7298992410809266176 to 7298992410653310976. Current Snapshot: MvccSnapshot[applied={T|T < 7298992410809266176}]
I20260620 18:36:09.064730 27314 mvcc.cc:205] Tried to move back new op lower bound from 7298992410809266176 to 7298992410653310976. Current Snapshot: MvccSnapshot[applied={T|T < 7298992410809266176}]
I20260620 18:36:09.074219 26836 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:36:09.074512 26836 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:36:09.078118 26836 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c: Sending DeleteTablet for 3 replicas of tablet e4b94d4b8eea4de491878f088c00f451
I20260620 18:36:09.079331 27219 tablet_service.cc:1514] Processing DeleteTablet for tablet e4b94d4b8eea4de491878f088c00f451 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:09 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:54970
I20260620 18:36:09.079344 26956 tablet_service.cc:1514] Processing DeleteTablet for tablet e4b94d4b8eea4de491878f088c00f451 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:09 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:33498
I20260620 18:36:09.079849 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 26873
I20260620 18:36:09.080046 27088 tablet_service.cc:1514] Processing DeleteTablet for tablet e4b94d4b8eea4de491878f088c00f451 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:09 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:50466
I20260620 18:36:09.082563 27323 tablet_replica.cc:330] T e4b94d4b8eea4de491878f088c00f451 P e0201587fac944f7a411a55384847506: stopping tablet replica
I20260620 18:36:09.083105 27323 raft_consensus.cc:2229] T e4b94d4b8eea4de491878f088c00f451 P e0201587fac944f7a411a55384847506 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:36:09.083650 27323 raft_consensus.cc:2258] T e4b94d4b8eea4de491878f088c00f451 P e0201587fac944f7a411a55384847506 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:09.084554 27325 tablet_replica.cc:330] T e4b94d4b8eea4de491878f088c00f451 P 41e3a33ef4884818ab9abd62b74412ac: stopping tablet replica
I20260620 18:36:09.085064 27325 raft_consensus.cc:2229] T e4b94d4b8eea4de491878f088c00f451 P 41e3a33ef4884818ab9abd62b74412ac [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:36:09.085888 27325 raft_consensus.cc:2258] T e4b94d4b8eea4de491878f088c00f451 P 41e3a33ef4884818ab9abd62b74412ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:09.086681 27324 tablet_replica.cc:330] T e4b94d4b8eea4de491878f088c00f451 P f28b4329d91c44b98ef80b977708d167: stopping tablet replica
I20260620 18:36:09.087110 27324 raft_consensus.cc:2229] T e4b94d4b8eea4de491878f088c00f451 P f28b4329d91c44b98ef80b977708d167 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:36:09.087659 27324 raft_consensus.cc:2258] T e4b94d4b8eea4de491878f088c00f451 P f28b4329d91c44b98ef80b977708d167 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:09.088642 27323 ts_tablet_manager.cc:1905] T e4b94d4b8eea4de491878f088c00f451 P e0201587fac944f7a411a55384847506: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:09.089670 27324 ts_tablet_manager.cc:1905] T e4b94d4b8eea4de491878f088c00f451 P f28b4329d91c44b98ef80b977708d167: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:09.095574 27323 ts_tablet_manager.cc:1918] T e4b94d4b8eea4de491878f088c00f451 P e0201587fac944f7a411a55384847506: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:36:09.095796 27323 log.cc:1196] T e4b94d4b8eea4de491878f088c00f451 P e0201587fac944f7a411a55384847506: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/wal/wals/e4b94d4b8eea4de491878f088c00f451
I20260620 18:36:09.096318 27323 ts_tablet_manager.cc:1939] T e4b94d4b8eea4de491878f088c00f451 P e0201587fac944f7a411a55384847506: Deleting consensus metadata
I20260620 18:36:09.096716 27324 ts_tablet_manager.cc:1918] T e4b94d4b8eea4de491878f088c00f451 P f28b4329d91c44b98ef80b977708d167: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:36:09.096966 27324 log.cc:1196] T e4b94d4b8eea4de491878f088c00f451 P f28b4329d91c44b98ef80b977708d167: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/wal/wals/e4b94d4b8eea4de491878f088c00f451
I20260620 18:36:09.097532 27324 ts_tablet_manager.cc:1939] T e4b94d4b8eea4de491878f088c00f451 P f28b4329d91c44b98ef80b977708d167: Deleting consensus metadata
I20260620 18:36:09.098287 26820 catalog_manager.cc:4841] TS e0201587fac944f7a411a55384847506 (127.10.131.3:33301): tablet e4b94d4b8eea4de491878f088c00f451 (table test-table [id=2db015a3431c4e48b942ccc98eadcb84]) successfully deleted
I20260620 18:36:09.099293 26820 catalog_manager.cc:4841] TS f28b4329d91c44b98ef80b977708d167 (127.10.131.1:40323): tablet e4b94d4b8eea4de491878f088c00f451 (table test-table [id=2db015a3431c4e48b942ccc98eadcb84]) successfully deleted
I20260620 18:36:09.100062 27325 ts_tablet_manager.cc:1905] T e4b94d4b8eea4de491878f088c00f451 P 41e3a33ef4884818ab9abd62b74412ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:09.100411 27299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:09.081234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33508 (local address 127.10.131.1:40323)
0620 18:36:09.081449 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:09.081475 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:09.081499 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:36:09.081572 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:09.081580 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:09.081679 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:09.081809 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:09.083775 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:09.084973 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:09.086962 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:09.087727 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:09.087864 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:09.092131 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:09.092148 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:09.092167 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:09.092211 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:09.096524 (+ 4313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:09.097540 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:09.097549 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:09.097553 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:09.097720 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:09.098215 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:09.098221 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:09.098224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:09.098744 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:09.099077 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:09.099809 (+ 732us) server_negotiation.cc:299] Negotiation successful
0620 18:36:09.100032 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:36:09.101158 27016 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:09.108613 27325 ts_tablet_manager.cc:1918] T e4b94d4b8eea4de491878f088c00f451 P 41e3a33ef4884818ab9abd62b74412ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:36:09.108814 27325 log.cc:1196] T e4b94d4b8eea4de491878f088c00f451 P 41e3a33ef4884818ab9abd62b74412ac: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/wal/wals/e4b94d4b8eea4de491878f088c00f451
I20260620 18:36:09.109403 27325 ts_tablet_manager.cc:1939] T e4b94d4b8eea4de491878f088c00f451 P 41e3a33ef4884818ab9abd62b74412ac: Deleting consensus metadata
I20260620 18:36:09.111454 26821 catalog_manager.cc:4841] TS 41e3a33ef4884818ab9abd62b74412ac (127.10.131.2:40545): tablet e4b94d4b8eea4de491878f088c00f451 (table test-table [id=2db015a3431c4e48b942ccc98eadcb84]) successfully deleted
I20260620 18:36:09.195691 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 26873
I20260620 18:36:09.219522 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 27025
I20260620 18:36:09.237251 27297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:09.220414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50480 (local address 127.10.131.2:40545)
0620 18:36:09.220601 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:09.220606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:09.220626 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:36:09.220821 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:09.220827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:09.220948 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:09.221057 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:09.222948 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:09.224255 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:09.226310 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:09.226953 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:09.227085 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:09.230485 (+ 3400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:09.230495 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:09.230508 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:09.230544 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:09.234088 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:09.234931 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:09.234937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:09.234939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:09.235082 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:09.235517 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:09.235520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:09.235522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:09.235901 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:09.236230 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:09.236582 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 18:36:09.236808 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 18:36:09.238098 27148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:09.357684 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 27025
I20260620 18:36:09.376638 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 27156
I20260620 18:36:09.389936 27295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:09.377486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54974 (local address 127.10.131.3:33301)
0620 18:36:09.377676 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:09.377680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:09.377698 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:36:09.377851 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:09.377855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:09.377930 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:09.378012 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:09.379482 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:09.380162 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:09.381596 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:09.382052 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:09.382144 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:09.384483 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:09.384496 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:09.384511 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:09.384546 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:09.387381 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:09.388036 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:09.388043 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:09.388045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:09.388188 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:09.388514 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:09.388517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:09.388519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:09.388926 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:09.389131 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:09.389389 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 18:36:09.389542 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:36:09.390548 27279 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:09.479913 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 27156
I20260620 18:36:09.496245 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 26803
I20260620 18:36:09.497459 26866 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:09.609012 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 26803
I20260620 18:36:29.632748 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:38121
--webserver_interface=127.10.131.62
--webserver_port=33489
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:37841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:38121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:29.956243 27330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:29.956609 27330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:29.956784 27330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:29.963981 27330 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 18:36:29.964116 27330 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 18:36:29.964208 27330 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 18:36:29.964299 27330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:36:29.964389 27330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:29.964488 27330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:29.964578 27330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:36:29.964656 27330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:36:29.974985 27330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:37841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:38121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:38121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=33489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:36:29.976267 27330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:29.977717 27330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:29.989398 27335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:29.989466 27338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:29.989527 27336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:29.990298 27330 server_base.cc:1034] running on GCE node
I20260620 18:36:29.991135 27330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:29.992451 27330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:29.993688 27330 hybrid_clock.cc:648] HybridClock initialized: now 1781980589993608 us; error 56 us; skew 500 ppm
I20260620 18:36:29.994019 27330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:29 dist-test-slave-cbn3 krb5kdc[26253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980589, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:30.001622 27330 init.cc:377] Logged in from keytab as kudu/127.10.131.62@KRBTEST.COM (short username kudu)
I20260620 18:36:30.003372 27330 webserver.cc:460] Webserver started at http://127.10.131.62:33489/ using document root <none> and password file <none>
I20260620 18:36:30.004154 27330 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:30.004297 27330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:30.010852 27330 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 18:36:30.014202 27345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:30.015677 27330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:36:30.015892 27330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "219191504a5243ac8cc1fcb988930d8c"
format_stamp: "Formatted at 2026-06-20 18:35:44 on dist-test-slave-cbn3"
server_key: "741e9131d0eb1922876e15143dd5e7ef"
server_key_iv: "8e9e30de5f3617c4417b9744be3b87b2"
server_key_version: "encryptionkey@0"
I20260620 18:36:30.016943 27330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:30.036618 27330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:30.040791 27330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:30.041136 27330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:30.060443 27330 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:38121
I20260620 18:36:30.060494 27396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:38121 every 8 connection(s)
I20260620 18:36:30.061983 27330 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:36:30.065275 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 27330
I20260620 18:36:30.066357 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.1:40323
--local_ip_for_outbound_sockets=127.10.131.1
--tserver_master_addrs=127.10.131.62:38121
--webserver_port=37115
--webserver_interface=127.10.131.1
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:37841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:36:30.070734 27397 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:36:30.075762 27397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c: Bootstrap starting.
I20260620 18:36:30.095878 27397 log.cc:824] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:30.111572 27397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:36:30.112113 27397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c: Bootstrap complete.
I20260620 18:36:30.121322 27397 raft_consensus.cc:348] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219191504a5243ac8cc1fcb988930d8c" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38121 } }
I20260620 18:36:30.122355 27397 raft_consensus.cc:729] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 219191504a5243ac8cc1fcb988930d8c, State: Initialized, Role: FOLLOWER
I20260620 18:36:30.122993 27397 consensus_queue.cc:260] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219191504a5243ac8cc1fcb988930d8c" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38121 } }
I20260620 18:36:30.123217 27397 raft_consensus.cc:388] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:36:30.123328 27397 raft_consensus.cc:482] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:36:30.123522 27397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [term 2 FOLLOWER]: Advancing to term 3
I20260620 18:36:30.127302 27397 raft_consensus.cc:504] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219191504a5243ac8cc1fcb988930d8c" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38121 } }
I20260620 18:36:30.127768 27397 leader_election.cc:302] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 219191504a5243ac8cc1fcb988930d8c; no voters:
I20260620 18:36:30.128430 27397 leader_election.cc:288] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 18:36:30.128569 27401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [term 3 FOLLOWER]: Leader election won for term 3
I20260620 18:36:30.129650 27401 raft_consensus.cc:686] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [term 3 LEADER]: Becoming Leader. State: Replica: 219191504a5243ac8cc1fcb988930d8c, State: Running, Role: LEADER
I20260620 18:36:30.130188 27401 consensus_queue.cc:237] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219191504a5243ac8cc1fcb988930d8c" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38121 } }
I20260620 18:36:30.130590 27397 sys_catalog.cc:564] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:36:30.132501 27403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 219191504a5243ac8cc1fcb988930d8c. Latest consensus state: current_term: 3 leader_uuid: "219191504a5243ac8cc1fcb988930d8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219191504a5243ac8cc1fcb988930d8c" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38121 } } }
I20260620 18:36:30.133091 27402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "219191504a5243ac8cc1fcb988930d8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219191504a5243ac8cc1fcb988930d8c" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38121 } } }
I20260620 18:36:30.133389 27402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:30.133811 27403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:30.143142 27407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:36:30.153689 27407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:36:30.155627 27407 catalog_manager.cc:1259] Loaded cluster ID: b6c54873d26c4dc996c1efb81e4b15b9
I20260620 18:36:30.155750 27407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:36:30.160885 27407 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:36:30.162925 27407 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c: Loaded TSK: 0
I20260620 18:36:30.163175 27407 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 219191504a5243ac8cc1fcb988930d8c: Loaded TSK: 1
F20260620 18:36:30.164558 27407 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f5a799659ad google::LogMessage::Fail() at ??:0
@ 0x7f5a7996888a google::LogMessage::SendToLog() at ??:0
I20260620 18:36:30.170325 27418 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f5a799654de google::LogMessage::Flush() at ??:0
@ 0x7f5a79966a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5a7d6c97d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f5a7d6b873e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f5a8f3941c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f5a8f393783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f5a8f41f2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f5a8f41f0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f5a8f4695e9 std::function<>::operator()() at ??:0
@ 0x7f5a8f397cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f5a8f395c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f5a8f41d610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f5a8f41d439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f5a7c5b1345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f5a7c5b491c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f5a7c5b4799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f5a7c58feaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f5a7f8696db start_thread at ??:0
@ 0x7f5a787ff61f clone at ??:0
W20260620 18:36:30.452152 27399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:30.452483 27399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:30.452677 27399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:30.460053 27399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:30.460216 27399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:30.460422 27399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.1
I20260620 18:36:30.470140 27399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:37841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.1:40323
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.131.1
--webserver_port=37115
--tserver_master_addrs=127.10.131.62:38121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.1
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:36:30.471565 27399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:30.472990 27399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:30.486423 27447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:30.486450 27445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:30.486786 27444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:30.487257 27399 server_base.cc:1034] running on GCE node
I20260620 18:36:30.487857 27399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:30.488984 27399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:30.490198 27399 hybrid_clock.cc:648] HybridClock initialized: now 1781980590490152 us; error 44 us; skew 500 ppm
I20260620 18:36:30.490545 27399 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:30 dist-test-slave-cbn3 krb5kdc[26253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980590, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:30.493901 27399 init.cc:377] Logged in from keytab as kudu/127.10.131.1@KRBTEST.COM (short username kudu)
I20260620 18:36:30.495390 27399 webserver.cc:460] Webserver started at http://127.10.131.1:37115/ using document root <none> and password file <none>
I20260620 18:36:30.496032 27399 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:30.496133 27399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:30.502000 27399 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 18:36:30.505024 27454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:30.506461 27399 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:36:30.506685 27399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/wal
uuid: "f28b4329d91c44b98ef80b977708d167"
format_stamp: "Formatted at 2026-06-20 18:35:44 on dist-test-slave-cbn3"
server_key: "5b37b6e9eb0024f30f53b378fd98782f"
server_key_iv: "9c0378c9b550135a00bc9d1cdb6ee6dd"
server_key_version: "encryptionkey@0"
I20260620 18:36:30.508029 27399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:30.549680 27399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:30.554332 27399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:30.554633 27399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:30.555773 27399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:30.557212 27399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:30.557337 27399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:30.557474 27399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:30.557562 27399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:30.591696 27399 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.1:40323
I20260620 18:36:30.591784 27566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.1:40323 every 8 connection(s)
I20260620 18:36:30.593348 27399 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
I20260620 18:36:30.598170 27569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:30.596200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.1:37995)
0620 18:36:30.597002 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:30.597245 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":275,"threads_started":1}
W20260620 18:36:30.598799 27567 heartbeater.cc:644] Failed to heartbeat to 127.10.131.62:38121 (0 consecutive failures): Network error: Failed to ping master at 127.10.131.62:38121: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
I20260620 18:36:30.599679 27569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:30.599348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.1:34303)
0620 18:36:30.599498 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:30.599577 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 18:36:30.600489 27569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:30.600182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.1:49589)
0620 18:36:30.600307 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:30.600376 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 18:36:30.600719 27567 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 18:36:30.603681 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 27399
I20260620 18:36:30.604442 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.2:40545
--local_ip_for_outbound_sockets=127.10.131.2
--tserver_master_addrs=127.10.131.62:38121
--webserver_port=33655
--webserver_interface=127.10.131.2
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:37841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:30.935770 27570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:30.936061 27570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:30.936160 27570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:30.943351 27570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:30.943459 27570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:30.943571 27570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.2
I20260620 18:36:30.953140 27570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:37841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.2:40545
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.131.2
--webserver_port=33655
--tserver_master_addrs=127.10.131.62:38121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.2
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:36:30.954336 27570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:30.955613 27570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:30.968573 27576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:30.968598 27578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:30.968647 27575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:30.969292 27570 server_base.cc:1034] running on GCE node
I20260620 18:36:30.969835 27570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:30.970685 27570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:30.971891 27570 hybrid_clock.cc:648] HybridClock initialized: now 1781980590971841 us; error 40 us; skew 500 ppm
I20260620 18:36:30.972226 27570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:30 dist-test-slave-cbn3 krb5kdc[26253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980590, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:30.975262 27570 init.cc:377] Logged in from keytab as kudu/127.10.131.2@KRBTEST.COM (short username kudu)
I20260620 18:36:30.976704 27570 webserver.cc:460] Webserver started at http://127.10.131.2:33655/ using document root <none> and password file <none>
I20260620 18:36:30.977424 27570 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:30.977571 27570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:30.982818 27570 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 18:36:30.985455 27585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:30.986711 27570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:36:30.986886 27570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/wal
uuid: "41e3a33ef4884818ab9abd62b74412ac"
format_stamp: "Formatted at 2026-06-20 18:35:45 on dist-test-slave-cbn3"
server_key: "5b944fe391bc39257d5882455852ec03"
server_key_iv: "6268bea237db663cdf38df60554e3ab5"
server_key_version: "encryptionkey@0"
I20260620 18:36:30.987856 27570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:31.006672 27570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:31.010701 27570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:31.010983 27570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:31.012077 27570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:31.013613 27570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:31.013731 27570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:31.013876 27570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:31.013969 27570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:31.047041 27570 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.2:40545
I20260620 18:36:31.047062 27697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.2:40545 every 8 connection(s)
I20260620 18:36:31.048484 27570 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
I20260620 18:36:31.053068 27700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:31.051301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.2:41363)
0620 18:36:31.051992 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:31.052215 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":260,"threads_started":1}
W20260620 18:36:31.053795 27698 heartbeater.cc:644] Failed to heartbeat to 127.10.131.62:38121 (0 consecutive failures): Network error: Failed to ping master at 127.10.131.62:38121: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
I20260620 18:36:31.054585 27700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:31.054277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.2:39061)
0620 18:36:31.054412 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:31.054481 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 18:36:31.055419 27700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:31.055131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.2:51357)
0620 18:36:31.055241 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:31.055311 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 18:36:31.055660 27698 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 18:36:31.057528 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 27570
I20260620 18:36:31.058138 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.3:33301
--local_ip_for_outbound_sockets=127.10.131.3
--tserver_master_addrs=127.10.131.62:38121
--webserver_port=44637
--webserver_interface=127.10.131.3
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:37841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:31.376951 27701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:31.377261 27701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:31.377391 27701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:31.384552 27701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:31.384658 27701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:31.384771 27701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.3
I20260620 18:36:31.394497 27701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:37841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.3:33301
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.131.3
--webserver_port=44637
--tserver_master_addrs=127.10.131.62:38121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.3
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:36:31.395694 27701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:31.397004 27701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:31.410334 27706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:31.410197 27707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:31.410343 27709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:31.411422 27701 server_base.cc:1034] running on GCE node
I20260620 18:36:31.412065 27701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:31.413164 27701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:31.414357 27701 hybrid_clock.cc:648] HybridClock initialized: now 1781980591414312 us; error 35 us; skew 500 ppm
I20260620 18:36:31.414680 27701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:31 dist-test-slave-cbn3 krb5kdc[26253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980591, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:31.417892 27701 init.cc:377] Logged in from keytab as kudu/127.10.131.3@KRBTEST.COM (short username kudu)
I20260620 18:36:31.419327 27701 webserver.cc:460] Webserver started at http://127.10.131.3:44637/ using document root <none> and password file <none>
I20260620 18:36:31.420045 27701 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:31.420174 27701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:31.425544 27701 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:36:31.428371 27716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:31.429756 27701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:36:31.429942 27701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/wal
uuid: "e0201587fac944f7a411a55384847506"
format_stamp: "Formatted at 2026-06-20 18:35:45 on dist-test-slave-cbn3"
server_key: "0c42b0ce96c7bc82cae239b8700e7d1a"
server_key_iv: "4a2bcd7a747c20d3e1a8653578c680d5"
server_key_version: "encryptionkey@0"
I20260620 18:36:31.430964 27701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:31.451272 27701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:31.455606 27701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:31.455897 27701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:31.456938 27701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:31.458268 27701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:31.458427 27701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:31.458567 27701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:31.458654 27701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:31.492208 27701 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.3:33301
I20260620 18:36:31.492242 27828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.3:33301 every 8 connection(s)
I20260620 18:36:31.493913 27701 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
I20260620 18:36:31.498730 27831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:31.496939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.3:60737)
0620 18:36:31.497653 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:31.497909 (+ 256us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":252,"threads_started":1}
W20260620 18:36:31.499392 27829 heartbeater.cc:644] Failed to heartbeat to 127.10.131.62:38121 (0 consecutive failures): Network error: Failed to ping master at 127.10.131.62:38121: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
I20260620 18:36:31.500221 27831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:31.499896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.3:43951)
0620 18:36:31.500044 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:31.500120 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 18:36:31.501036 27831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:31.500704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.3:60633)
0620 18:36:31.500857 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:31.500938 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 18:36:31.501304 27829 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 18:36:31.501349 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 27701
I20260620 18:36:31.602128 27833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:31.601304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.1:56923)
0620 18:36:31.601794 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:31.601906 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":218,"threads_started":1}
I20260620 18:36:32.057013 27834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:32.056172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.2:33423)
0620 18:36:32.056614 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:32.056720 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":211,"threads_started":1}
I20260620 18:36:32.502606 27835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:32.501817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.3:55851)
0620 18:36:32.502304 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:32.502388 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":272,"threads_started":1}
I20260620 18:36:32.603734 27836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:32.602962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.1:49583)
0620 18:36:32.603481 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:32.603567 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":218,"threads_started":1}
I20260620 18:36:33.058560 27837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:33.057888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.2:48197)
0620 18:36:33.058318 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:33.058398 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":212,"threads_started":1}
I20260620 18:36:33.504281 27838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:33.503584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.3:41151)
0620 18:36:33.504036 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:33.504115 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":217,"threads_started":1}
I20260620 18:36:33.605351 27839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:33.604571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.1:43695)
0620 18:36:33.605072 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:33.605158 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":281,"threads_started":1}
I20260620 18:36:34.060072 27840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:34.059388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.2:33723)
0620 18:36:34.059830 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:34.059910 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":228,"threads_started":1}
I20260620 18:36:34.505842 27841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:34.505173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.3:42761)
0620 18:36:34.505585 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:34.505665 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":212,"threads_started":1}
I20260620 18:36:34.606875 27842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:34.606187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.1:46455)
0620 18:36:34.606614 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:34.606694 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":231,"threads_started":1}
I20260620 18:36:35.061749 27843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:35.061010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.2:46243)
0620 18:36:35.061463 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:35.061544 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":234,"threads_started":1}
I20260620 18:36:35.507500 27844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:35.506815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.3:39057)
0620 18:36:35.507236 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:35.507316 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":217,"threads_started":1}
I20260620 18:36:35.608949 27845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:35.607903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.1:44047)
0620 18:36:35.608558 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:35.608680 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":329,"threads_started":1}
I20260620 18:36:36.063486 27846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:36.062729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.2:46353)
0620 18:36:36.063237 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:36.063321 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":245,"threads_started":1}
I20260620 18:36:36.509534 27847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:36.508668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.3:42415)
0620 18:36:36.509245 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:36.509345 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":345,"threads_started":1}
I20260620 18:36:36.610733 27848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:36.609912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.1:32783)
0620 18:36:36.610440 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:36.610542 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":290,"threads_started":1}
I20260620 18:36:37.065106 27849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:37.064344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.2:53503)
0620 18:36:37.064796 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:37.064945 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":246,"threads_started":1}
I20260620 18:36:37.511458 27850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:37.510511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.3:59453)
0620 18:36:37.511080 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:37.511201 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":290,"threads_started":1}
I20260620 18:36:37.612964 27851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:37.611791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.1:57765)
0620 18:36:37.612438 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:37.612575 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":276,"threads_started":1}
I20260620 18:36:38.066664 27852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:38.065929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.2:43885)
0620 18:36:38.066377 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:38.066459 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":199,"threads_started":1}
I20260620 18:36:38.513197 27853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:38.512309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.3:51595)
0620 18:36:38.512833 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:38.512966 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":224,"threads_started":1}
I20260620 18:36:38.614593 27854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:38.613866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.1:36277)
0620 18:36:38.614332 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:38.614432 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":249,"threads_started":1}
I20260620 18:36:39.068224 27855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:39.067557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.2:58939)
0620 18:36:39.067988 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:39.068068 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":214,"threads_started":1}
I20260620 18:36:39.514742 27856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:39.514073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.3:60733)
0620 18:36:39.514499 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:39.514581 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":213,"threads_started":1}
I20260620 18:36:39.616146 27857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:39.615432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.1:34137)
0620 18:36:39.615889 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:39.615978 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":219,"threads_started":1}
I20260620 18:36:40.069972 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:40.069154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.2:36051)
0620 18:36:40.069682 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:40.069775 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":262,"threads_started":1}
I20260620 18:36:40.516327 27859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:40.515596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.3:34455)
0620 18:36:40.516065 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:40.516149 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":234,"threads_started":1}
I20260620 18:36:40.617730 27860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:40.617054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.1:50149)
0620 18:36:40.617494 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:40.617571 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":246,"threads_started":1}
I20260620 18:36:41.071552 27861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:41.070891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.2:58741)
0620 18:36:41.071324 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:41.071401 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":228,"threads_started":1}
I20260620 18:36:41.518074 27862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:41.517383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.3:52443)
0620 18:36:41.517835 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:41.517915 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":245,"threads_started":1}
I20260620 18:36:41.619271 27863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:41.618590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.1:37179)
0620 18:36:41.618994 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:41.619084 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":202,"threads_started":1}
I20260620 18:36:42.073243 27864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:42.072423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.2:50345)
0620 18:36:42.072956 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:42.073056 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":239,"threads_started":1}
I20260620 18:36:42.519902 27865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:42.519025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.3:44359)
0620 18:36:42.519629 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:42.519727 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":383,"threads_started":1}
I20260620 18:36:42.620705 27866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:42.620078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.1:59481)
0620 18:36:42.620471 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:42.620549 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":201,"threads_started":1}
I20260620 18:36:43.074764 27867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:43.074130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.2:43143)
0620 18:36:43.074531 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:43.074609 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":205,"threads_started":1}
I20260620 18:36:43.521492 27868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:43.520795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.3:36011)
0620 18:36:43.521262 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:43.521342 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":261,"threads_started":1}
I20260620 18:36:43.622313 27869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:43.621589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.1:59219)
0620 18:36:43.622034 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:43.622124 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":239,"threads_started":1}
I20260620 18:36:44.076697 27870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:44.075799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.2:55983)
0620 18:36:44.076381 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:44.076485 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":275,"threads_started":1}
I20260620 18:36:44.523092 27871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:44.522384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.3:52425)
0620 18:36:44.522826 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:44.522914 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":222,"threads_started":1}
I20260620 18:36:44.623801 27872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:44.623152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.1:52717)
0620 18:36:44.623551 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:44.623635 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":210,"threads_started":1}
I20260620 18:36:45.078217 27873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:45.077532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.2:46581)
0620 18:36:45.077951 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:45.078039 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":215,"threads_started":1}
I20260620 18:36:45.524678 27874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:45.523980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.3:58735)
0620 18:36:45.524419 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:45.524503 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":228,"threads_started":1}
I20260620 18:36:45.625948 27875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:45.625106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.1:60039)
0620 18:36:45.625623 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:45.625737 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":261,"threads_started":1}
I20260620 18:36:46.079864 27876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.079175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.2:41959)
0620 18:36:46.079603 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:46.079687 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":212,"threads_started":1}
I20260620 18:36:46.506928 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 27399
I20260620 18:36:46.526401 27877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.508020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53422 (local address 127.10.131.1:40323)
0620 18:36:46.508957 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:46.508978 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:46.509025 (+ 47us) server_negotiation.cc:407] Connection header received
0620 18:36:46.510683 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:46.510692 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:46.511104 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:46.511451 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:46.513055 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.514255 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.515553 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.516243 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.516453 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:46.519150 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:46.519159 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:46.519175 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:46.519214 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:46.522382 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:46.523249 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:46.523259 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:46.523275 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:46.523472 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:46.524269 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:46.524276 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:46.524279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:46.524817 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:46.525222 (+ 405us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:46.525694 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 18:36:46.526121 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":545,"threads_started":1}
I20260620 18:36:46.527846 27878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.525842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38121 (local address 127.10.131.3:37177)
0620 18:36:46.526476 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:46.526594 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.62:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":330,"threads_started":1}
I20260620 18:36:46.528033 27562 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:46.615381 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 27399
I20260620 18:36:46.632567 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 27570
I20260620 18:36:46.650906 27880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.633622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37528 (local address 127.10.131.2:40545)
0620 18:36:46.634163 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:46.634192 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:46.634258 (+ 66us) server_negotiation.cc:407] Connection header received
0620 18:36:46.635836 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:46.635846 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:46.636299 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:46.636614 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:46.638171 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.639418 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.640762 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.641575 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.641735 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:46.645008 (+ 3273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:46.645020 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:46.645039 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:46.645084 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:46.647844 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:46.648528 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:46.648543 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:46.648550 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:46.648709 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:46.649113 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:46.649116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:46.649118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:46.649658 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:46.649876 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:46.650189 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 18:36:46.650621 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":272,"threads_started":1}
I20260620 18:36:46.652264 27693 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:46.736192 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 27570
I20260620 18:36:46.753708 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 27701
I20260620 18:36:46.771080 27882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:46.754632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47982 (local address 127.10.131.3:33301)
0620 18:36:46.755261 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:46.755275 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:46.755315 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:36:46.756962 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:46.756972 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:46.757397 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:46.757680 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:46.758890 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.760237 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.761676 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:46.762284 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:46.762410 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:46.764965 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:46.764974 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:46.764989 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:46.765026 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:46.767956 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:46.768597 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:46.768603 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:46.768610 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:46.768774 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:46.769261 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:46.769265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:46.769267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:46.769769 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:46.769984 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:46.770346 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 18:36:46.770756 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":311,"threads_started":1}
I20260620 18:36:46.772354 27824 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:46.854146 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 27701
2026-06-20T18:36:46Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64229 ms)
[----------] 30 tests from SecurityITest (292722 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:36:46 dist-test-slave-cbn3 krb5kdc[27886](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:36:46 dist-test-slave-cbn3 krb5kdc[27886](info): set up 2 sockets
Jun 20 18:36:46 dist-test-slave-cbn3 krb5kdc[27886](info): commencing operation
krb5kdc: starting...
W20260620 18:36:47.952975 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:36:48 dist-test-slave-cbn3 krb5kdc[27886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:36:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:36:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.131.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
I20260620 18:36:48.191762 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:34007
--webserver_interface=127.10.131.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:42897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:34007
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:48.520411 27899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:48.520813 27899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:48.521014 27899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:48.528297 27899 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:36:48.528462 27899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:48.528573 27899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:48.528661 27899 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:36:48.528728 27899 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:36:48.538661 27899 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:42897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:34007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:34007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:36:48.540004 27899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:48.541507 27899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:48.553503 27904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:48.553520 27907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:48.553550 27905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:48.554183 27899 server_base.cc:1034] running on GCE node
I20260620 18:36:48.554957 27899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:48.556241 27899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:48.557471 27899 hybrid_clock.cc:648] HybridClock initialized: now 1781980608557417 us; error 42 us; skew 500 ppm
I20260620 18:36:48.557834 27899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:48 dist-test-slave-cbn3 krb5kdc[27886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980608, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:48.563928 27899 init.cc:377] Logged in from keytab as kudu/127.10.131.62@KRBTEST.COM (short username kudu)
I20260620 18:36:48.565604 27899 webserver.cc:460] Webserver started at http://127.10.131.62:32993/ using document root <none> and password file <none>
I20260620 18:36:48.566301 27899 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:48.566444 27899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:48.566744 27899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:48.569818 27899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "eda74813d1e0441a8fa194f2168f45a5"
format_stamp: "Formatted at 2026-06-20 18:36:48 on dist-test-slave-cbn3"
server_key: "4b81432138095eb2cadaed9a428349c6"
server_key_iv: "eab75406e3d3851a845d20b65d2c75cf"
server_key_version: "encryptionkey@0"
I20260620 18:36:48.570560 27899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "eda74813d1e0441a8fa194f2168f45a5"
format_stamp: "Formatted at 2026-06-20 18:36:48 on dist-test-slave-cbn3"
server_key: "4b81432138095eb2cadaed9a428349c6"
server_key_iv: "eab75406e3d3851a845d20b65d2c75cf"
server_key_version: "encryptionkey@0"
I20260620 18:36:48.575678 27899 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:36:48.578999 27914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:48.580246 27899 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 18:36:48.580423 27899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "eda74813d1e0441a8fa194f2168f45a5"
format_stamp: "Formatted at 2026-06-20 18:36:48 on dist-test-slave-cbn3"
server_key: "4b81432138095eb2cadaed9a428349c6"
server_key_iv: "eab75406e3d3851a845d20b65d2c75cf"
server_key_version: "encryptionkey@0"
I20260620 18:36:48.580616 27899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:48.608690 27899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:48.613184 27899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:48.613499 27899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:48.631902 27899 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:34007
I20260620 18:36:48.631922 27965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:34007 every 8 connection(s)
I20260620 18:36:48.633396 27899 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:36:48.636925 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 27899
I20260620 18:36:48.637147 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:36:48.637293 27966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:48.637979 10764 external_mini_cluster.cc:1428] Setting key 61ab690b12237498e0f0c7b068a963ec
I20260620 18:36:48.651841 27966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eda74813d1e0441a8fa194f2168f45a5: Bootstrap starting.
Jun 20 18:36:48 dist-test-slave-cbn3 krb5kdc[27886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:36:48.655220 27966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eda74813d1e0441a8fa194f2168f45a5: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:48.656288 27966 log.cc:824] T 00000000000000000000000000000000 P eda74813d1e0441a8fa194f2168f45a5: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:48.659039 27966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eda74813d1e0441a8fa194f2168f45a5: No bootstrap required, opened a new log
I20260620 18:36:48.660753 27969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:48.639997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44618 (local address 127.10.131.62:34007)
0620 18:36:48.640812 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:48.640823 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:48.640915 (+ 92us) server_negotiation.cc:407] Connection header received
0620 18:36:48.642441 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:48.642462 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:48.642823 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:48.643141 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:48.644983 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:48.646218 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:48.648103 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:48.648773 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:48.648980 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:48.653655 (+ 4675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:48.653663 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:48.653676 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:48.653721 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:48.656808 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:48.657572 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:48.657580 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:48.657595 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:48.657752 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:48.658139 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:48.658145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:48.658148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:48.658641 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:48.658944 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:48.659382 (+ 438us) server_negotiation.cc:299] Negotiation successful
0620 18:36:48.659817 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":385,"threads_started":1}
I20260620 18:36:48.667106 27966 raft_consensus.cc:348] T 00000000000000000000000000000000 P eda74813d1e0441a8fa194f2168f45a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eda74813d1e0441a8fa194f2168f45a5" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34007 } }
I20260620 18:36:48.667464 27966 raft_consensus.cc:374] T 00000000000000000000000000000000 P eda74813d1e0441a8fa194f2168f45a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:48.667567 27966 raft_consensus.cc:729] T 00000000000000000000000000000000 P eda74813d1e0441a8fa194f2168f45a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eda74813d1e0441a8fa194f2168f45a5, State: Initialized, Role: FOLLOWER
I20260620 18:36:48.668165 27966 consensus_queue.cc:260] T 00000000000000000000000000000000 P eda74813d1e0441a8fa194f2168f45a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eda74813d1e0441a8fa194f2168f45a5" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34007 } }
I20260620 18:36:48.668437 27966 raft_consensus.cc:388] T 00000000000000000000000000000000 P eda74813d1e0441a8fa194f2168f45a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:36:48.668570 27966 raft_consensus.cc:482] T 00000000000000000000000000000000 P eda74813d1e0441a8fa194f2168f45a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:36:48.668747 27966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eda74813d1e0441a8fa194f2168f45a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:48.671154 27966 raft_consensus.cc:504] T 00000000000000000000000000000000 P eda74813d1e0441a8fa194f2168f45a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eda74813d1e0441a8fa194f2168f45a5" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34007 } }
I20260620 18:36:48.671636 27966 leader_election.cc:302] T 00000000000000000000000000000000 P eda74813d1e0441a8fa194f2168f45a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eda74813d1e0441a8fa194f2168f45a5; no voters:
I20260620 18:36:48.672194 27966 leader_election.cc:288] T 00000000000000000000000000000000 P eda74813d1e0441a8fa194f2168f45a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:36:48.672322 27971 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eda74813d1e0441a8fa194f2168f45a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:36:48.673116 27971 raft_consensus.cc:686] T 00000000000000000000000000000000 P eda74813d1e0441a8fa194f2168f45a5 [term 1 LEADER]: Becoming Leader. State: Replica: eda74813d1e0441a8fa194f2168f45a5, State: Running, Role: LEADER
I20260620 18:36:48.673691 27971 consensus_queue.cc:237] T 00000000000000000000000000000000 P eda74813d1e0441a8fa194f2168f45a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eda74813d1e0441a8fa194f2168f45a5" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34007 } }
I20260620 18:36:48.674345 27966 sys_catalog.cc:564] T 00000000000000000000000000000000 P eda74813d1e0441a8fa194f2168f45a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:36:48.676416 27972 sys_catalog.cc:455] T 00000000000000000000000000000000 P eda74813d1e0441a8fa194f2168f45a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eda74813d1e0441a8fa194f2168f45a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eda74813d1e0441a8fa194f2168f45a5" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34007 } } }
I20260620 18:36:48.676966 27972 sys_catalog.cc:458] T 00000000000000000000000000000000 P eda74813d1e0441a8fa194f2168f45a5 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:48.679776 27979 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:36:48.679702 27973 sys_catalog.cc:455] T 00000000000000000000000000000000 P eda74813d1e0441a8fa194f2168f45a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eda74813d1e0441a8fa194f2168f45a5. Latest consensus state: current_term: 1 leader_uuid: "eda74813d1e0441a8fa194f2168f45a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eda74813d1e0441a8fa194f2168f45a5" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34007 } } }
I20260620 18:36:48.680022 27973 sys_catalog.cc:458] T 00000000000000000000000000000000 P eda74813d1e0441a8fa194f2168f45a5 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:48.685220 27979 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:36:48.693895 27979 catalog_manager.cc:1347] Generated new cluster ID: 18e8416462354477a12f63b219e92f65
I20260620 18:36:48.694022 27979 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:36:48.717595 27979 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:36:48.719189 27979 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:36:48.728084 27979 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eda74813d1e0441a8fa194f2168f45a5: Generated new TSK 0
I20260620 18:36:48.729142 27979 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.131.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab.
Entry for principal kudu/127.10.131.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab.
I20260620 18:36:48.801779 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.1:0
--local_ip_for_outbound_sockets=127.10.131.1
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:34007
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:42897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:49.144246 27992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:49.144511 27992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:49.144603 27992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:49.151808 27992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:49.151916 27992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:49.152027 27992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.1
I20260620 18:36:49.161988 27992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:42897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:34007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.1
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:36:49.163331 27992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:49.164546 27992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:49.178125 28000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:49.178231 27997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:49.178103 27998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:49.178561 27992 server_base.cc:1034] running on GCE node
I20260620 18:36:49.179448 27992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:49.180433 27992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:49.181694 27992 hybrid_clock.cc:648] HybridClock initialized: now 1781980609181640 us; error 36 us; skew 500 ppm
I20260620 18:36:49.182096 27992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:49 dist-test-slave-cbn3 krb5kdc[27886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980609, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:49.185636 27992 init.cc:377] Logged in from keytab as kudu/127.10.131.1@KRBTEST.COM (short username kudu)
I20260620 18:36:49.187160 27992 webserver.cc:460] Webserver started at http://127.10.131.1:39883/ using document root <none> and password file <none>
I20260620 18:36:49.187829 27992 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:49.187975 27992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:49.188309 27992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:49.191267 27992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-0/data/instance:
uuid: "984e3280804b45c1b05c3ba6cb06d6ce"
format_stamp: "Formatted at 2026-06-20 18:36:49 on dist-test-slave-cbn3"
server_key: "1157dee87a8b676bb40d141665047f23"
server_key_iv: "e66c3350ababf86fc99eb3039ec95fa5"
server_key_version: "encryptionkey@0"
I20260620 18:36:49.192044 27992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance:
uuid: "984e3280804b45c1b05c3ba6cb06d6ce"
format_stamp: "Formatted at 2026-06-20 18:36:49 on dist-test-slave-cbn3"
server_key: "1157dee87a8b676bb40d141665047f23"
server_key_iv: "e66c3350ababf86fc99eb3039ec95fa5"
server_key_version: "encryptionkey@0"
I20260620 18:36:49.197170 27992 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260620 18:36:49.200533 28007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:49.201877 27992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:36:49.202059 27992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-0/wal
uuid: "984e3280804b45c1b05c3ba6cb06d6ce"
format_stamp: "Formatted at 2026-06-20 18:36:49 on dist-test-slave-cbn3"
server_key: "1157dee87a8b676bb40d141665047f23"
server_key_iv: "e66c3350ababf86fc99eb3039ec95fa5"
server_key_version: "encryptionkey@0"
I20260620 18:36:49.202262 27992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:49.217744 27992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:49.222019 27992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:49.222371 27992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:49.223532 27992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:49.225087 27992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:49.225198 27992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:49.225323 27992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:49.225394 27992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:49.259063 27992 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.1:41903
I20260620 18:36:49.259105 28119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.1:41903 every 8 connection(s)
I20260620 18:36:49.260373 27992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
I20260620 18:36:49.269802 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 27992
I20260620 18:36:49.270058 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance
I20260620 18:36:49.270691 10764 external_mini_cluster.cc:1428] Setting key 3b7df4c250a14d419e273e3c4f2e5509
Jun 20 18:36:49 dist-test-slave-cbn3 krb5kdc[27886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980609, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:36:49.279920 28122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:49.262981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:55347 (local address 127.10.131.62:34007)
0620 18:36:49.263406 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:49.263422 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:49.264311 (+ 889us) server_negotiation.cc:407] Connection header received
0620 18:36:49.266127 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:49.266131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:49.266202 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:49.266299 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:49.267748 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:49.268456 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:49.270010 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:49.270478 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:49.270615 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:49.274070 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:49.274081 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:49.274085 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:49.274118 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:49.276820 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:49.277603 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:49.277608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:49.277612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:49.277738 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:49.278165 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:49.278170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:49.278174 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:49.278440 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:49.278595 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:49.279303 (+ 708us) server_negotiation.cc:299] Negotiation successful
0620 18:36:49.279544 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":210,"threads_started":1}
I20260620 18:36:49.280916 28123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:49.263473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:34007 (local address 127.10.131.1:55347)
0620 18:36:49.264154 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:49.264192 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:49.265869 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:49.266578 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:49.266586 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:49.267063 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:49.267602 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:49.267618 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:49.268584 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:49.268588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:49.269848 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:49.269860 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:49.270716 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:49.270725 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:49.270966 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:49.271637 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:49.271660 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:49.273839 (+ 2179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:49.277017 (+ 3178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:49.277027 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:49.277045 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:49.277479 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:49.277847 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:49.277852 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:49.277856 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:49.278049 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:49.278567 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:49.278575 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:49.278931 (+ 356us) client_negotiation.cc:769] Sending connection context
0620 18:36:49.279195 (+ 264us) client_negotiation.cc:240] Negotiation successful
0620 18:36:49.279759 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":269,"threads_started":1}
I20260620 18:36:49.282514 28120 heartbeater.cc:347] Connected to a master server at 127.10.131.62:34007
I20260620 18:36:49.282837 28120 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:49.283560 28120 heartbeater.cc:511] Master 127.10.131.62:34007 requested a full tablet report, sending...
I20260620 18:36:49.285686 27931 ts_manager.cc:194] Registered new tserver with Master: 984e3280804b45c1b05c3ba6cb06d6ce (127.10.131.1:41903)
I20260620 18:36:49.287096 27931 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.1@KRBTEST.COM'} at 127.10.131.1:55347
WARNING: no policy specified for kudu/127.10.131.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab.
Entry for principal kudu/127.10.131.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab.
I20260620 18:36:49.322079 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.2:0
--local_ip_for_outbound_sockets=127.10.131.2
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:34007
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:42897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:49.658004 28126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:49.658289 28126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:49.658385 28126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:49.665654 28126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:49.665759 28126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:49.665876 28126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.2
I20260620 18:36:49.675870 28126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:42897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:34007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.2
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:36:49.677152 28126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:49.678452 28126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:49.692014 28131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:49.692013 28132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:49.692162 28134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:49.693390 28126 server_base.cc:1034] running on GCE node
I20260620 18:36:49.693949 28126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:49.694896 28126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:49.696120 28126 hybrid_clock.cc:648] HybridClock initialized: now 1781980609696085 us; error 47 us; skew 500 ppm
I20260620 18:36:49.696434 28126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:49 dist-test-slave-cbn3 krb5kdc[27886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980609, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:49.699420 28126 init.cc:377] Logged in from keytab as kudu/127.10.131.2@KRBTEST.COM (short username kudu)
I20260620 18:36:49.700976 28126 webserver.cc:460] Webserver started at http://127.10.131.2:39061/ using document root <none> and password file <none>
I20260620 18:36:49.701700 28126 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:49.701828 28126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:49.702154 28126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:49.705204 28126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-1/data/instance:
uuid: "a296e51e524847fd9e9b843d2e4145d2"
format_stamp: "Formatted at 2026-06-20 18:36:49 on dist-test-slave-cbn3"
server_key: "dcb56e3309c85b3cff2dc5b7a93e1eec"
server_key_iv: "37c81baa16d55a98440ce219649869a4"
server_key_version: "encryptionkey@0"
I20260620 18:36:49.705974 28126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance:
uuid: "a296e51e524847fd9e9b843d2e4145d2"
format_stamp: "Formatted at 2026-06-20 18:36:49 on dist-test-slave-cbn3"
server_key: "dcb56e3309c85b3cff2dc5b7a93e1eec"
server_key_iv: "37c81baa16d55a98440ce219649869a4"
server_key_version: "encryptionkey@0"
I20260620 18:36:49.710952 28126 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:36:49.714169 28141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:49.715524 28126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:36:49.715688 28126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-1/wal
uuid: "a296e51e524847fd9e9b843d2e4145d2"
format_stamp: "Formatted at 2026-06-20 18:36:49 on dist-test-slave-cbn3"
server_key: "dcb56e3309c85b3cff2dc5b7a93e1eec"
server_key_iv: "37c81baa16d55a98440ce219649869a4"
server_key_version: "encryptionkey@0"
I20260620 18:36:49.715896 28126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:49.729936 28126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:49.733779 28126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:49.734097 28126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:49.735013 28126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:49.736340 28126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:49.736415 28126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:49.736560 28126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:49.736611 28126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:49.769054 28126 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.2:34483
I20260620 18:36:49.769104 28253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.2:34483 every 8 connection(s)
I20260620 18:36:49.770360 28126 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
I20260620 18:36:49.776247 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 28126
I20260620 18:36:49.776461 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance
I20260620 18:36:49.777081 10764 external_mini_cluster.cc:1428] Setting key f69f441923e27116d507ef9d831434c6
Jun 20 18:36:49 dist-test-slave-cbn3 krb5kdc[27886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980609, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:36:49.792385 28122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:49.772755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:56015 (local address 127.10.131.62:34007)
0620 18:36:49.772958 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:49.772962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:49.774082 (+ 1120us) server_negotiation.cc:407] Connection header received
0620 18:36:49.776023 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:49.776028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:49.776123 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:49.776232 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:49.778182 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:49.779223 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:49.781787 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:49.782442 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:49.782596 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:49.786658 (+ 4062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:49.786663 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:49.786667 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:49.786698 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:49.789417 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:49.790122 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:49.790134 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:49.790136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:49.790256 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:49.790693 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:49.790698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:49.790701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:49.790961 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:49.791114 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:49.791802 (+ 688us) server_negotiation.cc:299] Negotiation successful
0620 18:36:49.792038 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:36:49.793462 28256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:49.773241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:34007 (local address 127.10.131.2:56015)
0620 18:36:49.773916 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:49.773960 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:49.775701 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:49.776577 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:49.776592 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:49.777325 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:49.777978 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:49.778002 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:49.779400 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:49.779405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:49.781590 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:49.781609 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:49.782702 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:49.782712 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:49.783058 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:49.783799 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:49.783833 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:49.786423 (+ 2590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:49.789566 (+ 3143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:49.789575 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:49.789592 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:49.789996 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:49.790365 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:49.790371 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:49.790375 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:49.790575 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:49.791086 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:49.791093 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:49.791446 (+ 353us) client_negotiation.cc:769] Sending connection context
0620 18:36:49.791721 (+ 275us) client_negotiation.cc:240] Negotiation successful
0620 18:36:49.792244 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":267,"threads_started":1}
I20260620 18:36:49.794955 28254 heartbeater.cc:347] Connected to a master server at 127.10.131.62:34007
I20260620 18:36:49.795261 28254 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:49.796041 28254 heartbeater.cc:511] Master 127.10.131.62:34007 requested a full tablet report, sending...
I20260620 18:36:49.797569 27931 ts_manager.cc:194] Registered new tserver with Master: a296e51e524847fd9e9b843d2e4145d2 (127.10.131.2:34483)
I20260620 18:36:49.798849 27931 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.2@KRBTEST.COM'} at 127.10.131.2:56015
WARNING: no policy specified for kudu/127.10.131.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab.
Entry for principal kudu/127.10.131.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab.
I20260620 18:36:49.829903 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.3:0
--local_ip_for_outbound_sockets=127.10.131.3
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:34007
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:42897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:50.149762 28259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:50.150040 28259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:50.150214 28259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:50.157480 28259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:50.157589 28259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:50.157799 28259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.3
I20260620 18:36:50.167428 28259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:42897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:34007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.3
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:36:50.168959 28259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:50.170275 28259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:50.183401 28267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:50.183494 28265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:50.183475 28264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:50.184343 28259 server_base.cc:1034] running on GCE node
I20260620 18:36:50.185010 28259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:50.186075 28259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:50.187287 28259 hybrid_clock.cc:648] HybridClock initialized: now 1781980610187246 us; error 35 us; skew 500 ppm
I20260620 18:36:50.187611 28259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:50 dist-test-slave-cbn3 krb5kdc[27886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980610, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:50.190780 28259 init.cc:377] Logged in from keytab as kudu/127.10.131.3@KRBTEST.COM (short username kudu)
I20260620 18:36:50.192328 28259 webserver.cc:460] Webserver started at http://127.10.131.3:33187/ using document root <none> and password file <none>
I20260620 18:36:50.193089 28259 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:50.193239 28259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:50.193559 28259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:50.196451 28259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-2/data/instance:
uuid: "1096a48952d948c2bfe9d321699ace31"
format_stamp: "Formatted at 2026-06-20 18:36:50 on dist-test-slave-cbn3"
server_key: "b82ebcf4f87e1f7c6ca811f2921d51e2"
server_key_iv: "22e3f767fa59582a576e1b6a10f912d0"
server_key_version: "encryptionkey@0"
I20260620 18:36:50.197268 28259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance:
uuid: "1096a48952d948c2bfe9d321699ace31"
format_stamp: "Formatted at 2026-06-20 18:36:50 on dist-test-slave-cbn3"
server_key: "b82ebcf4f87e1f7c6ca811f2921d51e2"
server_key_iv: "22e3f767fa59582a576e1b6a10f912d0"
server_key_version: "encryptionkey@0"
I20260620 18:36:50.202399 28259 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 18:36:50.205886 28274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:50.207317 28259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 18:36:50.207484 28259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-2/wal
uuid: "1096a48952d948c2bfe9d321699ace31"
format_stamp: "Formatted at 2026-06-20 18:36:50 on dist-test-slave-cbn3"
server_key: "b82ebcf4f87e1f7c6ca811f2921d51e2"
server_key_iv: "22e3f767fa59582a576e1b6a10f912d0"
server_key_version: "encryptionkey@0"
I20260620 18:36:50.207684 28259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:50.232512 28259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:50.236266 28259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:50.236591 28259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:50.237605 28259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:50.238857 28259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:50.238927 28259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:50.239090 28259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:50.239147 28259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:50.272294 28259 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.3:36979
I20260620 18:36:50.272410 28386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.3:36979 every 8 connection(s)
I20260620 18:36:50.273658 28259 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
I20260620 18:36:50.282877 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 28259
I20260620 18:36:50.283090 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance
I20260620 18:36:50.283715 10764 external_mini_cluster.cc:1428] Setting key 920496ded254355646823bd8b8377bc8
Jun 20 18:36:50 dist-test-slave-cbn3 krb5kdc[27886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980610, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:36:50.290005 28120 heartbeater.cc:503] Master 127.10.131.62:34007 was elected leader, sending a full tablet report...
I20260620 18:36:50.293720 28122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:50.276139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:47795 (local address 127.10.131.62:34007)
0620 18:36:50.276352 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:50.276358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:50.277480 (+ 1122us) server_negotiation.cc:407] Connection header received
0620 18:36:50.279185 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:50.279189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:50.279257 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:50.279342 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:50.280680 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.281533 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.283522 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.283975 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.284071 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:50.287710 (+ 3639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:50.287714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:50.287717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:50.287750 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:50.290684 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:50.291400 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:50.291406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:50.291410 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:50.291578 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:50.291979 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:50.291986 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:50.291989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:50.292269 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:50.292424 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:50.293158 (+ 734us) server_negotiation.cc:299] Negotiation successful
0620 18:36:50.293365 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 18:36:50.294651 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:50.276608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:34007 (local address 127.10.131.3:47795)
0620 18:36:50.277323 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:50.277362 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:50.278948 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:50.279529 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:50.279536 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:50.280018 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:50.280535 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:50.280550 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.281688 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.281692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:50.283360 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:50.283376 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.284192 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.284202 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:50.284508 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:50.285294 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:50.285326 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:50.287529 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:50.290824 (+ 3295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:50.290834 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:50.290849 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:50.291277 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:50.291677 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:50.291682 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:50.291687 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:50.291868 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:50.292400 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:50.292407 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:50.292719 (+ 312us) client_negotiation.cc:769] Sending connection context
0620 18:36:50.293062 (+ 343us) client_negotiation.cc:240] Negotiation successful
0620 18:36:50.293563 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":317,"threads_started":1}
I20260620 18:36:50.295836 28387 heartbeater.cc:347] Connected to a master server at 127.10.131.62:34007
I20260620 18:36:50.296118 28387 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:50.296764 28387 heartbeater.cc:511] Master 127.10.131.62:34007 requested a full tablet report, sending...
I20260620 18:36:50.298144 27931 ts_manager.cc:194] Registered new tserver with Master: 1096a48952d948c2bfe9d321699ace31 (127.10.131.3:36979)
I20260620 18:36:50.299196 10764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:36:50.299238 27931 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.3@KRBTEST.COM'} at 127.10.131.3:47795
I20260620 18:36:50.315651 28122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:50.303600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44628 (local address 127.10.131.62:34007)
0620 18:36:50.303776 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:50.303780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:50.304233 (+ 453us) server_negotiation.cc:407] Connection header received
0620 18:36:50.304354 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:50.304357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:50.304412 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:50.304487 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:50.306094 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.306845 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.308210 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.308585 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.308670 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:50.310895 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:50.310900 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:50.310902 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:50.310929 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:50.313559 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:50.314131 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:50.314134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:50.314136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:50.314251 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:50.314556 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:50.314559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:50.314561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:50.314834 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:50.315019 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:50.315278 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 18:36:50.315427 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:36:50.319739 27931 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:36:50.328651 27931 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:36:50.358806 28402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:50.341879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57208 (local address 127.10.131.3:36979)
0620 18:36:50.350433 (+ 8554us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:50.350442 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:50.350473 (+ 31us) server_negotiation.cc:407] Connection header received
0620 18:36:50.350576 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:50.350586 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:50.351233 (+ 647us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:50.352139 (+ 906us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:50.352411 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.353728 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.356174 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.357614 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.357753 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:50.358002 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:50.358270 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 18:36:50.358455 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8370,"thread_start_us":8404,"threads_started":1}
I20260620 18:36:50.358848 28401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:50.341369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:36979 (local address 127.0.0.1:57208)
0620 18:36:50.350085 (+ 8716us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:50.350112 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:50.350227 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:50.351465 (+ 1238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:50.351472 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:50.351512 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:50.351935 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:50.351947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.353856 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.353861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:50.356002 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:50.356016 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.357815 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.357822 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:50.358025 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:50.358074 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 18:36:50.358215 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 18:36:50.358411 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8584,"mutex_wait_us":104,"thread_start_us":8398,"threads_started":1}
I20260620 18:36:50.359645 28399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:50.340791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:34483 (local address 127.0.0.1:35510)
0620 18:36:50.347364 (+ 6573us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:50.347417 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:50.347588 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:50.352111 (+ 4523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:50.352115 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:50.352139 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:50.352500 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:50.352515 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.354559 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.354564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:50.356774 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:50.356785 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.358571 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.358577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:50.358724 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:50.358759 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 18:36:50.358884 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 18:36:50.359005 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":64,"thread_start_us":484,"threads_started":1}
I20260620 18:36:50.359666 28400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:50.340898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35510 (local address 127.10.131.2:34483)
0620 18:36:50.350897 (+ 9999us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:50.350905 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:50.350931 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:36:50.351019 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:50.351025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:50.351216 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:50.352679 (+ 1463us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:50.352935 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.354423 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.356999 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.358150 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.358268 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:50.358427 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:50.359146 (+ 719us) server_negotiation.cc:299] Negotiation successful
0620 18:36:50.359317 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9843,"thread_start_us":9709,"threads_started":1}
I20260620 18:36:50.359680 28403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:50.342104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:41903 (local address 127.0.0.1:42314)
0620 18:36:50.351284 (+ 9180us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:50.351311 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:50.351450 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:50.352631 (+ 1181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:50.352636 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:50.352663 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:50.353020 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:50.353032 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.354816 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.354823 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:50.356697 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:50.356710 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.358813 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.358820 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:50.359018 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:50.359053 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 18:36:50.359143 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 18:36:50.359295 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9017,"mutex_wait_us":7,"thread_start_us":8884,"threads_started":1}
I20260620 18:36:50.360512 28404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:50.342814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42314 (local address 127.10.131.1:41903)
0620 18:36:50.352037 (+ 9223us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:50.352048 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:50.352074 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:36:50.352163 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:50.352173 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:50.352427 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:50.352626 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:50.353223 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.354683 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.356917 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.358308 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.358445 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:50.358652 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:50.359805 (+ 1153us) server_negotiation.cc:299] Negotiation successful
0620 18:36:50.359973 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9064,"thread_start_us":8904,"threads_started":1}
I20260620 18:36:50.363379 28189 tablet_service.cc:1467] Processing CreateTablet for tablet cb47bd239cfb4f1b8e1a40b54c83b5dc (DEFAULT_TABLE table=test-table [id=158bcd312dd04776b36eab08768bda52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:50.363304 28322 tablet_service.cc:1467] Processing CreateTablet for tablet cb47bd239cfb4f1b8e1a40b54c83b5dc (DEFAULT_TABLE table=test-table [id=158bcd312dd04776b36eab08768bda52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:50.363569 28055 tablet_service.cc:1467] Processing CreateTablet for tablet cb47bd239cfb4f1b8e1a40b54c83b5dc (DEFAULT_TABLE table=test-table [id=158bcd312dd04776b36eab08768bda52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:50.364718 28055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb47bd239cfb4f1b8e1a40b54c83b5dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:50.364753 28322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb47bd239cfb4f1b8e1a40b54c83b5dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:50.364763 28189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb47bd239cfb4f1b8e1a40b54c83b5dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:50.376195 28405 tablet_bootstrap.cc:492] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 984e3280804b45c1b05c3ba6cb06d6ce: Bootstrap starting.
I20260620 18:36:50.380640 28405 tablet_bootstrap.cc:654] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 984e3280804b45c1b05c3ba6cb06d6ce: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:50.381493 28407 tablet_bootstrap.cc:492] T cb47bd239cfb4f1b8e1a40b54c83b5dc P a296e51e524847fd9e9b843d2e4145d2: Bootstrap starting.
I20260620 18:36:50.382045 28405 log.cc:824] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 984e3280804b45c1b05c3ba6cb06d6ce: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:50.383661 28406 tablet_bootstrap.cc:492] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 1096a48952d948c2bfe9d321699ace31: Bootstrap starting.
I20260620 18:36:50.385339 28407 tablet_bootstrap.cc:654] T cb47bd239cfb4f1b8e1a40b54c83b5dc P a296e51e524847fd9e9b843d2e4145d2: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:50.385382 28405 tablet_bootstrap.cc:492] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 984e3280804b45c1b05c3ba6cb06d6ce: No bootstrap required, opened a new log
I20260620 18:36:50.385668 28405 ts_tablet_manager.cc:1397] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 984e3280804b45c1b05c3ba6cb06d6ce: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 18:36:50.386584 28407 log.cc:824] T cb47bd239cfb4f1b8e1a40b54c83b5dc P a296e51e524847fd9e9b843d2e4145d2: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:50.387174 28406 tablet_bootstrap.cc:654] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 1096a48952d948c2bfe9d321699ace31: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:50.388351 28406 log.cc:824] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 1096a48952d948c2bfe9d321699ace31: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:50.389920 28407 tablet_bootstrap.cc:492] T cb47bd239cfb4f1b8e1a40b54c83b5dc P a296e51e524847fd9e9b843d2e4145d2: No bootstrap required, opened a new log
I20260620 18:36:50.390229 28407 ts_tablet_manager.cc:1397] T cb47bd239cfb4f1b8e1a40b54c83b5dc P a296e51e524847fd9e9b843d2e4145d2: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260620 18:36:50.393816 28405 raft_consensus.cc:348] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 984e3280804b45c1b05c3ba6cb06d6ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a296e51e524847fd9e9b843d2e4145d2" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 34483 } } peers { permanent_uuid: "984e3280804b45c1b05c3ba6cb06d6ce" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41903 } } peers { permanent_uuid: "1096a48952d948c2bfe9d321699ace31" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 36979 } }
I20260620 18:36:50.394169 28405 raft_consensus.cc:374] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 984e3280804b45c1b05c3ba6cb06d6ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:50.394289 28405 raft_consensus.cc:729] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 984e3280804b45c1b05c3ba6cb06d6ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 984e3280804b45c1b05c3ba6cb06d6ce, State: Initialized, Role: FOLLOWER
I20260620 18:36:50.394899 28405 consensus_queue.cc:260] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 984e3280804b45c1b05c3ba6cb06d6ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a296e51e524847fd9e9b843d2e4145d2" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 34483 } } peers { permanent_uuid: "984e3280804b45c1b05c3ba6cb06d6ce" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41903 } } peers { permanent_uuid: "1096a48952d948c2bfe9d321699ace31" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 36979 } }
I20260620 18:36:50.395041 28406 tablet_bootstrap.cc:492] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 1096a48952d948c2bfe9d321699ace31: No bootstrap required, opened a new log
I20260620 18:36:50.395282 28406 ts_tablet_manager.cc:1397] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 1096a48952d948c2bfe9d321699ace31: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260620 18:36:50.396559 28405 ts_tablet_manager.cc:1428] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 984e3280804b45c1b05c3ba6cb06d6ce: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260620 18:36:50.401063 28407 raft_consensus.cc:348] T cb47bd239cfb4f1b8e1a40b54c83b5dc P a296e51e524847fd9e9b843d2e4145d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a296e51e524847fd9e9b843d2e4145d2" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 34483 } } peers { permanent_uuid: "984e3280804b45c1b05c3ba6cb06d6ce" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41903 } } peers { permanent_uuid: "1096a48952d948c2bfe9d321699ace31" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 36979 } }
I20260620 18:36:50.401568 28407 raft_consensus.cc:374] T cb47bd239cfb4f1b8e1a40b54c83b5dc P a296e51e524847fd9e9b843d2e4145d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:50.401700 28407 raft_consensus.cc:729] T cb47bd239cfb4f1b8e1a40b54c83b5dc P a296e51e524847fd9e9b843d2e4145d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a296e51e524847fd9e9b843d2e4145d2, State: Initialized, Role: FOLLOWER
I20260620 18:36:50.402346 28407 consensus_queue.cc:260] T cb47bd239cfb4f1b8e1a40b54c83b5dc P a296e51e524847fd9e9b843d2e4145d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a296e51e524847fd9e9b843d2e4145d2" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 34483 } } peers { permanent_uuid: "984e3280804b45c1b05c3ba6cb06d6ce" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41903 } } peers { permanent_uuid: "1096a48952d948c2bfe9d321699ace31" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 36979 } }
I20260620 18:36:50.403342 28254 heartbeater.cc:503] Master 127.10.131.62:34007 was elected leader, sending a full tablet report...
I20260620 18:36:50.404006 28407 ts_tablet_manager.cc:1428] T cb47bd239cfb4f1b8e1a40b54c83b5dc P a296e51e524847fd9e9b843d2e4145d2: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260620 18:36:50.405537 28412 raft_consensus.cc:482] T cb47bd239cfb4f1b8e1a40b54c83b5dc P a296e51e524847fd9e9b843d2e4145d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:36:50.405836 28412 raft_consensus.cc:504] T cb47bd239cfb4f1b8e1a40b54c83b5dc P a296e51e524847fd9e9b843d2e4145d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a296e51e524847fd9e9b843d2e4145d2" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 34483 } } peers { permanent_uuid: "984e3280804b45c1b05c3ba6cb06d6ce" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41903 } } peers { permanent_uuid: "1096a48952d948c2bfe9d321699ace31" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 36979 } }
I20260620 18:36:50.406963 28406 raft_consensus.cc:348] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 1096a48952d948c2bfe9d321699ace31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a296e51e524847fd9e9b843d2e4145d2" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 34483 } } peers { permanent_uuid: "984e3280804b45c1b05c3ba6cb06d6ce" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41903 } } peers { permanent_uuid: "1096a48952d948c2bfe9d321699ace31" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 36979 } }
I20260620 18:36:50.407390 28406 raft_consensus.cc:374] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 1096a48952d948c2bfe9d321699ace31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:50.407507 28406 raft_consensus.cc:729] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 1096a48952d948c2bfe9d321699ace31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1096a48952d948c2bfe9d321699ace31, State: Initialized, Role: FOLLOWER
I20260620 18:36:50.408095 28406 consensus_queue.cc:260] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 1096a48952d948c2bfe9d321699ace31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a296e51e524847fd9e9b843d2e4145d2" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 34483 } } peers { permanent_uuid: "984e3280804b45c1b05c3ba6cb06d6ce" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41903 } } peers { permanent_uuid: "1096a48952d948c2bfe9d321699ace31" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 36979 } }
I20260620 18:36:50.408890 28412 leader_election.cc:288] T cb47bd239cfb4f1b8e1a40b54c83b5dc P a296e51e524847fd9e9b843d2e4145d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 984e3280804b45c1b05c3ba6cb06d6ce (127.10.131.1:41903), 1096a48952d948c2bfe9d321699ace31 (127.10.131.3:36979)
I20260620 18:36:50.409917 28406 ts_tablet_manager.cc:1428] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 1096a48952d948c2bfe9d321699ace31: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260620 18:36:50.410370 28387 heartbeater.cc:503] Master 127.10.131.62:34007 was elected leader, sending a full tablet report...
I20260620 18:36:50.415906 28404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:50.407848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:54941 (local address 127.10.131.1:41903)
0620 18:36:50.408036 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:50.408042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:50.409034 (+ 992us) server_negotiation.cc:407] Connection header received
0620 18:36:50.409308 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:50.409315 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:50.409411 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:50.409517 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:50.410174 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.411288 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.413955 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.414955 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.415135 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:50.415210 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:50.415520 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 18:36:50.415643 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:36:50.415942 28413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:50.407771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:41903 (local address 127.10.131.2:54941)
0620 18:36:50.408930 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:50.408950 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:50.409146 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:50.409555 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:50.409559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:50.409584 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:50.409961 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:50.409975 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.411448 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.411455 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:50.413779 (+ 2324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:50.413791 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.415159 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.415165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:50.415364 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:50.415387 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 18:36:50.415485 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 18:36:50.415604 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"thread_start_us":620,"threads_started":1}
I20260620 18:36:50.416760 28402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:50.409801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:38705 (local address 127.10.131.3:36979)
0620 18:36:50.410017 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:50.410022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:50.410965 (+ 943us) server_negotiation.cc:407] Connection header received
0620 18:36:50.411154 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:50.411160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:50.411226 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:50.411329 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:50.411977 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.413147 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.414997 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.415753 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.415950 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:50.416022 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:50.416398 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 18:36:50.416516 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:36:50.416811 28415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:50.409257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:36979 (local address 127.10.131.2:38705)
0620 18:36:50.410829 (+ 1572us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:50.410856 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:50.410974 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:50.411358 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:50.411363 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:50.411388 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:50.411779 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:50.411788 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.413355 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.413361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:50.414856 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:50.414867 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.415993 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.415999 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:50.416234 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:50.416267 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:36:50.416373 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 18:36:50.416497 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1467,"thread_start_us":1292,"threads_started":1}
I20260620 18:36:50.416929 28075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb47bd239cfb4f1b8e1a40b54c83b5dc" candidate_uuid: "a296e51e524847fd9e9b843d2e4145d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "984e3280804b45c1b05c3ba6cb06d6ce" is_pre_election: true
I20260620 18:36:50.417518 28075 raft_consensus.cc:2454] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 984e3280804b45c1b05c3ba6cb06d6ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a296e51e524847fd9e9b843d2e4145d2 in term 0.
I20260620 18:36:50.417809 28342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb47bd239cfb4f1b8e1a40b54c83b5dc" candidate_uuid: "a296e51e524847fd9e9b843d2e4145d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1096a48952d948c2bfe9d321699ace31" is_pre_election: true
I20260620 18:36:50.418262 28142 leader_election.cc:302] T cb47bd239cfb4f1b8e1a40b54c83b5dc P a296e51e524847fd9e9b843d2e4145d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 984e3280804b45c1b05c3ba6cb06d6ce, a296e51e524847fd9e9b843d2e4145d2; no voters:
I20260620 18:36:50.418290 28342 raft_consensus.cc:2454] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 1096a48952d948c2bfe9d321699ace31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a296e51e524847fd9e9b843d2e4145d2 in term 0.
I20260620 18:36:50.418674 28412 raft_consensus.cc:2781] T cb47bd239cfb4f1b8e1a40b54c83b5dc P a296e51e524847fd9e9b843d2e4145d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:36:50.418824 28412 raft_consensus.cc:482] T cb47bd239cfb4f1b8e1a40b54c83b5dc P a296e51e524847fd9e9b843d2e4145d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:36:50.418936 28412 raft_consensus.cc:3037] T cb47bd239cfb4f1b8e1a40b54c83b5dc P a296e51e524847fd9e9b843d2e4145d2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:50.421185 28412 raft_consensus.cc:504] T cb47bd239cfb4f1b8e1a40b54c83b5dc P a296e51e524847fd9e9b843d2e4145d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a296e51e524847fd9e9b843d2e4145d2" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 34483 } } peers { permanent_uuid: "984e3280804b45c1b05c3ba6cb06d6ce" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41903 } } peers { permanent_uuid: "1096a48952d948c2bfe9d321699ace31" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 36979 } }
I20260620 18:36:50.421741 28412 leader_election.cc:288] T cb47bd239cfb4f1b8e1a40b54c83b5dc P a296e51e524847fd9e9b843d2e4145d2 [CANDIDATE]: Term 1 election: Requested vote from peers 984e3280804b45c1b05c3ba6cb06d6ce (127.10.131.1:41903), 1096a48952d948c2bfe9d321699ace31 (127.10.131.3:36979)
I20260620 18:36:50.422238 28075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb47bd239cfb4f1b8e1a40b54c83b5dc" candidate_uuid: "a296e51e524847fd9e9b843d2e4145d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "984e3280804b45c1b05c3ba6cb06d6ce"
I20260620 18:36:50.422281 28342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb47bd239cfb4f1b8e1a40b54c83b5dc" candidate_uuid: "a296e51e524847fd9e9b843d2e4145d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1096a48952d948c2bfe9d321699ace31"
I20260620 18:36:50.422459 28342 raft_consensus.cc:3037] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 1096a48952d948c2bfe9d321699ace31 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:50.422459 28075 raft_consensus.cc:3037] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 984e3280804b45c1b05c3ba6cb06d6ce [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:50.425630 28075 raft_consensus.cc:2454] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 984e3280804b45c1b05c3ba6cb06d6ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a296e51e524847fd9e9b843d2e4145d2 in term 1.
I20260620 18:36:50.425630 28342 raft_consensus.cc:2454] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 1096a48952d948c2bfe9d321699ace31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a296e51e524847fd9e9b843d2e4145d2 in term 1.
I20260620 18:36:50.426048 28145 leader_election.cc:302] T cb47bd239cfb4f1b8e1a40b54c83b5dc P a296e51e524847fd9e9b843d2e4145d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1096a48952d948c2bfe9d321699ace31, 984e3280804b45c1b05c3ba6cb06d6ce, a296e51e524847fd9e9b843d2e4145d2; no voters:
I20260620 18:36:50.426407 28412 raft_consensus.cc:2781] T cb47bd239cfb4f1b8e1a40b54c83b5dc P a296e51e524847fd9e9b843d2e4145d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:36:50.426833 28412 raft_consensus.cc:686] T cb47bd239cfb4f1b8e1a40b54c83b5dc P a296e51e524847fd9e9b843d2e4145d2 [term 1 LEADER]: Becoming Leader. State: Replica: a296e51e524847fd9e9b843d2e4145d2, State: Running, Role: LEADER
I20260620 18:36:50.427265 28412 consensus_queue.cc:237] T cb47bd239cfb4f1b8e1a40b54c83b5dc P a296e51e524847fd9e9b843d2e4145d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a296e51e524847fd9e9b843d2e4145d2" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 34483 } } peers { permanent_uuid: "984e3280804b45c1b05c3ba6cb06d6ce" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41903 } } peers { permanent_uuid: "1096a48952d948c2bfe9d321699ace31" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 36979 } }
I20260620 18:36:50.431725 27931 catalog_manager.cc:5495] T cb47bd239cfb4f1b8e1a40b54c83b5dc P a296e51e524847fd9e9b843d2e4145d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a296e51e524847fd9e9b843d2e4145d2 (127.10.131.2). New cstate: current_term: 1 leader_uuid: "a296e51e524847fd9e9b843d2e4145d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a296e51e524847fd9e9b843d2e4145d2" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 34483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "984e3280804b45c1b05c3ba6cb06d6ce" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1096a48952d948c2bfe9d321699ace31" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 36979 } health_report { overall_health: UNKNOWN } } }
I20260620 18:36:50.449000 28400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:50.443741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35526 (local address 127.10.131.2:34483)
0620 18:36:50.443953 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:50.443960 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:50.443978 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:36:50.444101 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:50.444105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:50.444173 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:50.444313 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:36:50.444687 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.445450 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.446919 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.447312 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.447447 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:50.447955 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:36:50.448413 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:36:50.448566 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:50.448643 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 18:36:50.448720 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:36:50.456564 28342 raft_consensus.cc:1261] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 1096a48952d948c2bfe9d321699ace31 [term 1 FOLLOWER]: Refusing update from remote peer a296e51e524847fd9e9b843d2e4145d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:36:50.456563 28075 raft_consensus.cc:1261] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 984e3280804b45c1b05c3ba6cb06d6ce [term 1 FOLLOWER]: Refusing update from remote peer a296e51e524847fd9e9b843d2e4145d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:36:50.457592 28416 consensus_queue.cc:1035] T cb47bd239cfb4f1b8e1a40b54c83b5dc P a296e51e524847fd9e9b843d2e4145d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "984e3280804b45c1b05c3ba6cb06d6ce" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:36:50.457999 28412 consensus_queue.cc:1035] T cb47bd239cfb4f1b8e1a40b54c83b5dc P a296e51e524847fd9e9b843d2e4145d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1096a48952d948c2bfe9d321699ace31" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 36979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:36:50.479538 28421 mvcc.cc:205] Tried to move back new op lower bound from 7298992580420358144 to 7298992580313944064. Current Snapshot: MvccSnapshot[applied={T|T < 7298992580420358144}]
I20260620 18:36:50.489619 27930 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:36:50.489944 27930 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:36:50.493738 27930 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eda74813d1e0441a8fa194f2168f45a5: Sending DeleteTablet for 3 replicas of tablet cb47bd239cfb4f1b8e1a40b54c83b5dc
I20260620 18:36:50.494802 28189 tablet_service.cc:1514] Processing DeleteTablet for tablet cb47bd239cfb4f1b8e1a40b54c83b5dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:50 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:35510
I20260620 18:36:50.494891 28055 tablet_service.cc:1514] Processing DeleteTablet for tablet cb47bd239cfb4f1b8e1a40b54c83b5dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:50 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:42314
I20260620 18:36:50.495535 28322 tablet_service.cc:1514] Processing DeleteTablet for tablet cb47bd239cfb4f1b8e1a40b54c83b5dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:50 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:57208
I20260620 18:36:50.497838 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 27992
I20260620 18:36:50.501462 28431 tablet_replica.cc:330] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 1096a48952d948c2bfe9d321699ace31: stopping tablet replica
I20260620 18:36:50.509438 28431 raft_consensus.cc:2229] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 1096a48952d948c2bfe9d321699ace31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:36:50.510203 28429 tablet_replica.cc:330] T cb47bd239cfb4f1b8e1a40b54c83b5dc P a296e51e524847fd9e9b843d2e4145d2: stopping tablet replica
I20260620 18:36:50.510681 28429 raft_consensus.cc:2229] T cb47bd239cfb4f1b8e1a40b54c83b5dc P a296e51e524847fd9e9b843d2e4145d2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:36:50.511269 28429 raft_consensus.cc:2258] T cb47bd239cfb4f1b8e1a40b54c83b5dc P a296e51e524847fd9e9b843d2e4145d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:50.512019 28431 raft_consensus.cc:2258] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 1096a48952d948c2bfe9d321699ace31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:50.513973 28429 ts_tablet_manager.cc:1905] T cb47bd239cfb4f1b8e1a40b54c83b5dc P a296e51e524847fd9e9b843d2e4145d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:50.517408 28430 tablet_replica.cc:330] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 984e3280804b45c1b05c3ba6cb06d6ce: stopping tablet replica
I20260620 18:36:50.518724 28431 ts_tablet_manager.cc:1905] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 1096a48952d948c2bfe9d321699ace31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:50.519171 28430 raft_consensus.cc:2229] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 984e3280804b45c1b05c3ba6cb06d6ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:36:50.519979 28429 ts_tablet_manager.cc:1918] T cb47bd239cfb4f1b8e1a40b54c83b5dc P a296e51e524847fd9e9b843d2e4145d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:36:50.520186 28429 log.cc:1196] T cb47bd239cfb4f1b8e1a40b54c83b5dc P a296e51e524847fd9e9b843d2e4145d2: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-1/wal/wals/cb47bd239cfb4f1b8e1a40b54c83b5dc
I20260620 18:36:50.520763 28429 ts_tablet_manager.cc:1939] T cb47bd239cfb4f1b8e1a40b54c83b5dc P a296e51e524847fd9e9b843d2e4145d2: Deleting consensus metadata
I20260620 18:36:50.522523 28430 raft_consensus.cc:2258] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 984e3280804b45c1b05c3ba6cb06d6ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:50.522562 27915 catalog_manager.cc:4841] TS a296e51e524847fd9e9b843d2e4145d2 (127.10.131.2:34483): tablet cb47bd239cfb4f1b8e1a40b54c83b5dc (table test-table [id=158bcd312dd04776b36eab08768bda52]) successfully deleted
Jun 20 18:36:50 dist-test-slave-cbn3 krb5kdc[27886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.1@KRBTEST.COM
I20260620 18:36:50.524478 28430 ts_tablet_manager.cc:1905] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 984e3280804b45c1b05c3ba6cb06d6ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:50.527510 28431 ts_tablet_manager.cc:1918] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 1096a48952d948c2bfe9d321699ace31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:36:50.527714 28431 log.cc:1196] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 1096a48952d948c2bfe9d321699ace31: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-2/wal/wals/cb47bd239cfb4f1b8e1a40b54c83b5dc
I20260620 18:36:50.528262 28431 ts_tablet_manager.cc:1939] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 1096a48952d948c2bfe9d321699ace31: Deleting consensus metadata
I20260620 18:36:50.530088 27918 catalog_manager.cc:4841] TS 1096a48952d948c2bfe9d321699ace31 (127.10.131.3:36979): tablet cb47bd239cfb4f1b8e1a40b54c83b5dc (table test-table [id=158bcd312dd04776b36eab08768bda52]) successfully deleted
I20260620 18:36:50.531536 28430 ts_tablet_manager.cc:1918] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 984e3280804b45c1b05c3ba6cb06d6ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:36:50.531769 28430 log.cc:1196] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 984e3280804b45c1b05c3ba6cb06d6ce: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781980314095189-10764-0/minicluster-data/ts-0/wal/wals/cb47bd239cfb4f1b8e1a40b54c83b5dc
I20260620 18:36:50.531786 28404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:50.499760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42320 (local address 127.10.131.1:41903)
0620 18:36:50.499982 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:50.499991 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:50.510109 (+ 10118us) server_negotiation.cc:407] Connection header received
0620 18:36:50.510336 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:50.510343 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:50.510465 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:50.510598 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:50.512743 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.514382 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.517533 (+ 3151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.518191 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.518314 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:50.524010 (+ 5696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:50.524025 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:50.524043 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:50.524092 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:50.528088 (+ 3996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:50.529044 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:50.529059 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:50.529062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:50.529224 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:50.529706 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:50.529712 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:50.529716 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:50.530222 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:50.530559 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:50.531086 (+ 527us) server_negotiation.cc:299] Negotiation successful
0620 18:36:50.531338 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:36:50.532352 28430 ts_tablet_manager.cc:1939] T cb47bd239cfb4f1b8e1a40b54c83b5dc P 984e3280804b45c1b05c3ba6cb06d6ce: Deleting consensus metadata
I20260620 18:36:50.532615 28115 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:50.670269 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 27992
I20260620 18:36:50.670444 27915 catalog_manager.cc:4841] TS 984e3280804b45c1b05c3ba6cb06d6ce (127.10.131.1:41903): tablet cb47bd239cfb4f1b8e1a40b54c83b5dc (table test-table [id=158bcd312dd04776b36eab08768bda52]) successfully deleted
I20260620 18:36:50.689057 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 28126
Jun 20 18:36:50 dist-test-slave-cbn3 krb5kdc[27886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.2@KRBTEST.COM
I20260620 18:36:50.705678 28400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:50.689859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35532 (local address 127.10.131.2:34483)
0620 18:36:50.690040 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:50.690044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:50.690127 (+ 83us) server_negotiation.cc:407] Connection header received
0620 18:36:50.690322 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:50.690326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:50.690392 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:50.690470 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:50.692097 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.693212 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.694534 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.694965 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.695118 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:50.699508 (+ 4390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:50.699521 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:50.699535 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:50.699575 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:50.702572 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:50.703298 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:50.703309 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:50.703318 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:50.703495 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:50.703847 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:50.703852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:50.703855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:50.704309 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:50.704633 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:50.704810 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 18:36:50.705072 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 18:36:50.706475 28249 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:50.818751 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 28126
I20260620 18:36:50.837075 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 28259
Jun 20 18:36:50 dist-test-slave-cbn3 krb5kdc[27886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.3@KRBTEST.COM
I20260620 18:36:50.851296 28402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:50.837797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57216 (local address 127.10.131.3:36979)
0620 18:36:50.837972 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:50.837977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:50.837994 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:36:50.838101 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:50.838105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:50.838168 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:50.838268 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:50.839463 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.840207 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.841540 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:50.841913 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:50.842002 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:50.846207 (+ 4205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:50.846219 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:50.846235 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:50.846263 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:50.849131 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:50.849720 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:50.849727 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:50.849729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:50.849847 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:50.850152 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:50.850155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:50.850157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:50.850477 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:50.850680 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:50.850900 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 18:36:50.851046 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:36:50.851913 28382 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:50.943611 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 28259
I20260620 18:36:50.959839 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 27899
I20260620 18:36:50.960979 27961 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:51.069275 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 27899
2026-06-20T18:36:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4217 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T18:36:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:36:51Z Disabled control of system clock
I20260620 18:36:51.143623 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:38085
--webserver_interface=127.10.131.62
--webserver_port=0
--builtin_ntp_servers=127.10.131.20:46819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:38085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:36:51.463027 28440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:51.463312 28440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:51.463393 28440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:51.470594 28440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:36:51.470722 28440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:51.470767 28440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:51.470818 28440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:36:51.470866 28440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:36:51.480288 28440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:46819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:38085
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:38085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:36:51.481583 28440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:51.483340 28440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:51.495183 28448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:51.495247 28446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:51.495244 28445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:51.496116 28440 server_base.cc:1034] running on GCE node
I20260620 18:36:51.496907 28440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:51.498312 28440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:51.499519 28440 hybrid_clock.cc:648] HybridClock initialized: now 1781980611499477 us; error 31 us; skew 500 ppm
I20260620 18:36:51.499876 28440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:36:51.502249 28440 webserver.cc:460] Webserver started at http://127.10.131.62:39737/ using document root <none> and password file <none>
I20260620 18:36:51.502933 28440 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:51.503017 28440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:51.503355 28440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:51.506515 28440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "eaf0d22b669641dd9e40b5f45a3603b4"
format_stamp: "Formatted at 2026-06-20 18:36:51 on dist-test-slave-cbn3"
server_key: "ee258b9d45200599555a8279a12e713b"
server_key_iv: "73b44cb8b82f103388a1367cc55c3f65"
server_key_version: "encryptionkey@0"
I20260620 18:36:51.507385 28440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "eaf0d22b669641dd9e40b5f45a3603b4"
format_stamp: "Formatted at 2026-06-20 18:36:51 on dist-test-slave-cbn3"
server_key: "ee258b9d45200599555a8279a12e713b"
server_key_iv: "73b44cb8b82f103388a1367cc55c3f65"
server_key_version: "encryptionkey@0"
I20260620 18:36:51.512692 28440 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:36:51.515997 28454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:51.517210 28440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:36:51.517407 28440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "eaf0d22b669641dd9e40b5f45a3603b4"
format_stamp: "Formatted at 2026-06-20 18:36:51 on dist-test-slave-cbn3"
server_key: "ee258b9d45200599555a8279a12e713b"
server_key_iv: "73b44cb8b82f103388a1367cc55c3f65"
server_key_version: "encryptionkey@0"
I20260620 18:36:51.517616 28440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:51.550545 28440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:51.551450 28440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:51.551738 28440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:51.569977 28440 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:38085
I20260620 18:36:51.569995 28505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:38085 every 8 connection(s)
I20260620 18:36:51.571370 28440 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:36:51.574728 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 28440
I20260620 18:36:51.574901 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:36:51.575409 28506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:51.575471 10764 external_mini_cluster.cc:1428] Setting key c40fa1b76f0a2fb37f70a8538b045b11
I20260620 18:36:51.588955 28509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:51.577309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44062 (local address 127.10.131.62:38085)
0620 18:36:51.578319 (+ 1010us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:51.578329 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:51.578368 (+ 39us) server_negotiation.cc:407] Connection header received
0620 18:36:51.579959 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:51.579982 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:51.580424 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:51.580720 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:51.581357 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.582644 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.584530 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:51.585264 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:51.585392 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:51.586653 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:51.586660 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:51.586672 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:51.587227 (+ 555us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:51.587304 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:51.587324 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:51.587427 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:51.587660 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 18:36:51.587984 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":371,"threads_started":1}
I20260620 18:36:51.589165 28506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaf0d22b669641dd9e40b5f45a3603b4: Bootstrap starting.
I20260620 18:36:51.592156 28506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eaf0d22b669641dd9e40b5f45a3603b4: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:51.593168 28506 log.cc:824] T 00000000000000000000000000000000 P eaf0d22b669641dd9e40b5f45a3603b4: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:51.595723 28506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaf0d22b669641dd9e40b5f45a3603b4: No bootstrap required, opened a new log
I20260620 18:36:51.602662 28506 raft_consensus.cc:348] T 00000000000000000000000000000000 P eaf0d22b669641dd9e40b5f45a3603b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf0d22b669641dd9e40b5f45a3603b4" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38085 } }
I20260620 18:36:51.603015 28506 raft_consensus.cc:374] T 00000000000000000000000000000000 P eaf0d22b669641dd9e40b5f45a3603b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:51.603122 28506 raft_consensus.cc:729] T 00000000000000000000000000000000 P eaf0d22b669641dd9e40b5f45a3603b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf0d22b669641dd9e40b5f45a3603b4, State: Initialized, Role: FOLLOWER
I20260620 18:36:51.603690 28506 consensus_queue.cc:260] T 00000000000000000000000000000000 P eaf0d22b669641dd9e40b5f45a3603b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf0d22b669641dd9e40b5f45a3603b4" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38085 } }
I20260620 18:36:51.603928 28506 raft_consensus.cc:388] T 00000000000000000000000000000000 P eaf0d22b669641dd9e40b5f45a3603b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:36:51.604084 28506 raft_consensus.cc:482] T 00000000000000000000000000000000 P eaf0d22b669641dd9e40b5f45a3603b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:36:51.604260 28506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eaf0d22b669641dd9e40b5f45a3603b4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:51.606442 28506 raft_consensus.cc:504] T 00000000000000000000000000000000 P eaf0d22b669641dd9e40b5f45a3603b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf0d22b669641dd9e40b5f45a3603b4" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38085 } }
I20260620 18:36:51.606921 28506 leader_election.cc:302] T 00000000000000000000000000000000 P eaf0d22b669641dd9e40b5f45a3603b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eaf0d22b669641dd9e40b5f45a3603b4; no voters:
I20260620 18:36:51.607456 28506 leader_election.cc:288] T 00000000000000000000000000000000 P eaf0d22b669641dd9e40b5f45a3603b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:36:51.607609 28511 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eaf0d22b669641dd9e40b5f45a3603b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:36:51.608771 28511 raft_consensus.cc:686] T 00000000000000000000000000000000 P eaf0d22b669641dd9e40b5f45a3603b4 [term 1 LEADER]: Becoming Leader. State: Replica: eaf0d22b669641dd9e40b5f45a3603b4, State: Running, Role: LEADER
I20260620 18:36:51.609464 28511 consensus_queue.cc:237] T 00000000000000000000000000000000 P eaf0d22b669641dd9e40b5f45a3603b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf0d22b669641dd9e40b5f45a3603b4" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38085 } }
I20260620 18:36:51.609735 28506 sys_catalog.cc:564] T 00000000000000000000000000000000 P eaf0d22b669641dd9e40b5f45a3603b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:36:51.615089 28512 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaf0d22b669641dd9e40b5f45a3603b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eaf0d22b669641dd9e40b5f45a3603b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf0d22b669641dd9e40b5f45a3603b4" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38085 } } }
I20260620 18:36:51.615543 28512 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaf0d22b669641dd9e40b5f45a3603b4 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:51.616123 28513 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaf0d22b669641dd9e40b5f45a3603b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eaf0d22b669641dd9e40b5f45a3603b4. Latest consensus state: current_term: 1 leader_uuid: "eaf0d22b669641dd9e40b5f45a3603b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf0d22b669641dd9e40b5f45a3603b4" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38085 } } }
I20260620 18:36:51.616662 28513 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaf0d22b669641dd9e40b5f45a3603b4 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:51.617822 28519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:36:51.622900 28519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:36:51.631652 28519 catalog_manager.cc:1347] Generated new cluster ID: 57fb644fe9f146a9ade55daa129efa7f
I20260620 18:36:51.631803 28519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:36:51.660789 28519 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:36:51.662189 28519 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:36:51.670239 28519 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eaf0d22b669641dd9e40b5f45a3603b4: Generated new TSK 0
I20260620 18:36:51.671037 28519 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:36:51.683975 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.1:0
--local_ip_for_outbound_sockets=127.10.131.1
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:38085
--builtin_ntp_servers=127.10.131.20:46819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:36:52.008221 28531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:52.008550 28531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:52.008718 28531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:52.016465 28531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:52.016609 28531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:52.016793 28531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.1
I20260620 18:36:52.026456 28531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:46819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:38085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.1
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:36:52.027755 28531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:52.029618 28531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:52.042506 28536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:52.042444 28537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:52.042479 28539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:52.043475 28531 server_base.cc:1034] running on GCE node
I20260620 18:36:52.044081 28531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:52.045040 28531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:52.046269 28531 hybrid_clock.cc:648] HybridClock initialized: now 1781980612046231 us; error 47 us; skew 500 ppm
I20260620 18:36:52.046590 28531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:36:52.048700 28531 webserver.cc:460] Webserver started at http://127.10.131.1:35185/ using document root <none> and password file <none>
I20260620 18:36:52.049427 28531 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:52.049562 28531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:52.049865 28531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:52.052603 28531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-0/data/instance:
uuid: "0c4e34f3900e458eaceb040161168c4f"
format_stamp: "Formatted at 2026-06-20 18:36:52 on dist-test-slave-cbn3"
server_key: "5c7565353aa618cc14873356b71f0208"
server_key_iv: "e910b48e555510d653fecf6b6d05df65"
server_key_version: "encryptionkey@0"
I20260620 18:36:52.053440 28531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance:
uuid: "0c4e34f3900e458eaceb040161168c4f"
format_stamp: "Formatted at 2026-06-20 18:36:52 on dist-test-slave-cbn3"
server_key: "5c7565353aa618cc14873356b71f0208"
server_key_iv: "e910b48e555510d653fecf6b6d05df65"
server_key_version: "encryptionkey@0"
I20260620 18:36:52.058403 28531 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:36:52.061885 28545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:52.063174 28531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:36:52.063352 28531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-0/wal
uuid: "0c4e34f3900e458eaceb040161168c4f"
format_stamp: "Formatted at 2026-06-20 18:36:52 on dist-test-slave-cbn3"
server_key: "5c7565353aa618cc14873356b71f0208"
server_key_iv: "e910b48e555510d653fecf6b6d05df65"
server_key_version: "encryptionkey@0"
I20260620 18:36:52.063587 28531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:52.081643 28531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:52.082413 28531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:52.082733 28531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:52.083676 28531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:52.085170 28531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:52.085300 28531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:52.085424 28531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:52.085512 28531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:52.120195 28531 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.1:42159
I20260620 18:36:52.120285 28657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.1:42159 every 8 connection(s)
I20260620 18:36:52.121740 28531 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
I20260620 18:36:52.127888 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 28531
I20260620 18:36:52.128118 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance
I20260620 18:36:52.128665 10764 external_mini_cluster.cc:1428] Setting key 765f4f1f108c32e63ead197c9d352822
I20260620 18:36:52.131356 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.2:0
--local_ip_for_outbound_sockets=127.10.131.2
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:38085
--builtin_ntp_servers=127.10.131.20:46819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:36:52.134982 28660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:52.124313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:59461 (local address 127.10.131.62:38085)
0620 18:36:52.124738 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:52.124744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:52.126073 (+ 1329us) server_negotiation.cc:407] Connection header received
0620 18:36:52.127784 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:52.127789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:52.127882 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:52.128052 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:52.129265 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:52.130182 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:52.131862 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:52.132627 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:52.132773 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:52.134062 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:52.134071 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:52.134078 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:52.134102 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:52.134152 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:52.134169 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:52.134260 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:52.134468 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 18:36:52.134572 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":195,"threads_started":1}
I20260620 18:36:52.135713 28661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:52.124976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38085 (local address 127.10.131.1:59461)
0620 18:36:52.125924 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:52.125968 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:52.127531 (+ 1563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:52.128338 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:52.128346 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:52.128425 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:52.129089 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:52.129106 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:52.130353 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:52.130359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:52.131727 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:52.131740 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:52.132818 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:52.132823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:52.133066 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:52.133749 (+ 683us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:36:52.133751 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:52.133789 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:36:52.133865 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:52.134263 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:52.134271 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:52.134276 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:36:52.134410 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 18:36:52.134691 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":239,"threads_started":1}
I20260620 18:36:52.137279 28658 heartbeater.cc:347] Connected to a master server at 127.10.131.62:38085
I20260620 18:36:52.137562 28658 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:52.138278 28658 heartbeater.cc:511] Master 127.10.131.62:38085 requested a full tablet report, sending...
I20260620 18:36:52.140389 28471 ts_manager.cc:194] Registered new tserver with Master: 0c4e34f3900e458eaceb040161168c4f (127.10.131.1:42159)
I20260620 18:36:52.142612 28471 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.131.1:59461
W20260620 18:36:52.471832 28662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:52.472131 28662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:52.472219 28662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:52.479590 28662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:52.479703 28662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:52.479812 28662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.2
I20260620 18:36:52.489171 28662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:46819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:38085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.2
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:36:52.490442 28662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:52.492594 28662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:52.505556 28670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:52.505674 28667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:52.505674 28668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:52.506686 28662 server_base.cc:1034] running on GCE node
I20260620 18:36:52.507354 28662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:52.508280 28662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:52.509523 28662 hybrid_clock.cc:648] HybridClock initialized: now 1781980612509486 us; error 46 us; skew 500 ppm
I20260620 18:36:52.509846 28662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:36:52.512056 28662 webserver.cc:460] Webserver started at http://127.10.131.2:46417/ using document root <none> and password file <none>
I20260620 18:36:52.512748 28662 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:52.512903 28662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:52.513206 28662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:52.515946 28662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-1/data/instance:
uuid: "8e2b25f838964e0790441b1f7499706a"
format_stamp: "Formatted at 2026-06-20 18:36:52 on dist-test-slave-cbn3"
server_key: "67b94e1170d2f3999e957dd438543259"
server_key_iv: "7938698c12e1c5e814a17501fcb73f1e"
server_key_version: "encryptionkey@0"
I20260620 18:36:52.516659 28662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance:
uuid: "8e2b25f838964e0790441b1f7499706a"
format_stamp: "Formatted at 2026-06-20 18:36:52 on dist-test-slave-cbn3"
server_key: "67b94e1170d2f3999e957dd438543259"
server_key_iv: "7938698c12e1c5e814a17501fcb73f1e"
server_key_version: "encryptionkey@0"
I20260620 18:36:52.521668 28662 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 18:36:52.524887 28676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:52.526232 28662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:36:52.526398 28662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-1/wal
uuid: "8e2b25f838964e0790441b1f7499706a"
format_stamp: "Formatted at 2026-06-20 18:36:52 on dist-test-slave-cbn3"
server_key: "67b94e1170d2f3999e957dd438543259"
server_key_iv: "7938698c12e1c5e814a17501fcb73f1e"
server_key_version: "encryptionkey@0"
I20260620 18:36:52.526638 28662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:52.557083 28662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:52.557965 28662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:52.558279 28662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:52.559258 28662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:52.560740 28662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:52.560882 28662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:52.561020 28662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:52.561138 28662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:52.593369 28662 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.2:34941
I20260620 18:36:52.593408 28788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.2:34941 every 8 connection(s)
I20260620 18:36:52.594794 28662 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
I20260620 18:36:52.596812 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 28662
I20260620 18:36:52.597028 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance
I20260620 18:36:52.597503 10764 external_mini_cluster.cc:1428] Setting key 4d93643b5af8d9b3b4bf57fe127e1873
I20260620 18:36:52.600371 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.3:0
--local_ip_for_outbound_sockets=127.10.131.3
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:38085
--builtin_ntp_servers=127.10.131.20:46819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:36:52.608078 28660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:52.597541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:51717 (local address 127.10.131.62:38085)
0620 18:36:52.597743 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:52.597749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:52.599364 (+ 1615us) server_negotiation.cc:407] Connection header received
0620 18:36:52.601317 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:52.601325 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:52.601417 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:52.601528 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:52.602492 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:52.603621 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:52.605158 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:52.605868 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:52.606007 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:52.607222 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:52.607231 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:52.607238 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:52.607262 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:52.607303 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:52.607321 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:52.607410 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:52.607593 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 18:36:52.607707 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:36:52.609267 28791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:52.598107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38085 (local address 127.10.131.2:51717)
0620 18:36:52.599211 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:52.599258 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:52.601070 (+ 1812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:52.601720 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:52.601727 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:52.601809 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:52.602331 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:52.602346 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:52.603769 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:52.603772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:52.605023 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:52.605035 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:52.606057 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:52.606061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:52.606245 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:52.606932 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:36:52.606934 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:52.606968 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:36:52.607036 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:52.607410 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:52.607416 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:52.607421 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:36:52.607549 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 18:36:52.607818 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":338,"threads_started":1}
I20260620 18:36:52.610704 28789 heartbeater.cc:347] Connected to a master server at 127.10.131.62:38085
I20260620 18:36:52.611030 28789 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:52.611765 28789 heartbeater.cc:511] Master 127.10.131.62:38085 requested a full tablet report, sending...
I20260620 18:36:52.613161 28471 ts_manager.cc:194] Registered new tserver with Master: 8e2b25f838964e0790441b1f7499706a (127.10.131.2:34941)
I20260620 18:36:52.613952 28471 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.131.2:51717
W20260620 18:36:52.916968 28792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:52.917235 28792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:52.917399 28792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:52.924779 28792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:52.924893 28792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:52.925074 28792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.3
I20260620 18:36:52.934079 28792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:46819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:38085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.3
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:36:52.935312 28792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:52.937016 28792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:52.949358 28798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:52.949414 28800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:52.949491 28797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:52.950323 28792 server_base.cc:1034] running on GCE node
I20260620 18:36:52.950937 28792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:52.951858 28792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:52.953068 28792 hybrid_clock.cc:648] HybridClock initialized: now 1781980612953021 us; error 35 us; skew 500 ppm
I20260620 18:36:52.953361 28792 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:36:52.955541 28792 webserver.cc:460] Webserver started at http://127.10.131.3:33451/ using document root <none> and password file <none>
I20260620 18:36:52.956198 28792 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:52.956283 28792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:52.956565 28792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:52.959472 28792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-2/data/instance:
uuid: "849c0a695b754bd990bc26a00a26cf8b"
format_stamp: "Formatted at 2026-06-20 18:36:52 on dist-test-slave-cbn3"
server_key: "64ba666036cfa59ad8994f3735249dda"
server_key_iv: "b7305ec8eea37a5507da0d6aab59ab7b"
server_key_version: "encryptionkey@0"
I20260620 18:36:52.960225 28792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance:
uuid: "849c0a695b754bd990bc26a00a26cf8b"
format_stamp: "Formatted at 2026-06-20 18:36:52 on dist-test-slave-cbn3"
server_key: "64ba666036cfa59ad8994f3735249dda"
server_key_iv: "b7305ec8eea37a5507da0d6aab59ab7b"
server_key_version: "encryptionkey@0"
I20260620 18:36:52.965370 28792 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:36:52.968931 28806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:52.970290 28792 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:36:52.970433 28792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-2/wal
uuid: "849c0a695b754bd990bc26a00a26cf8b"
format_stamp: "Formatted at 2026-06-20 18:36:52 on dist-test-slave-cbn3"
server_key: "64ba666036cfa59ad8994f3735249dda"
server_key_iv: "b7305ec8eea37a5507da0d6aab59ab7b"
server_key_version: "encryptionkey@0"
I20260620 18:36:52.970611 28792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:52.982694 28792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:52.983475 28792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:52.983747 28792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:52.984658 28792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:52.986176 28792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:52.986269 28792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:52.986358 28792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:52.986428 28792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:53.020905 28792 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.3:40617
I20260620 18:36:53.020939 28918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.3:40617 every 8 connection(s)
I20260620 18:36:53.022464 28792 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
I20260620 18:36:53.025913 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 28792
I20260620 18:36:53.030664 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance
I20260620 18:36:53.031340 10764 external_mini_cluster.cc:1428] Setting key 4e904c4a1ce58fb0f2b3651d1f0eb7f0
I20260620 18:36:53.037091 28660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:53.024977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:51005 (local address 127.10.131.62:38085)
0620 18:36:53.025168 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:53.025178 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:53.026857 (+ 1679us) server_negotiation.cc:407] Connection header received
0620 18:36:53.029852 (+ 2995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:53.029857 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:53.029929 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:53.030022 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:53.031537 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.032497 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.034638 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.035050 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.035146 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:53.036389 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:53.036393 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:53.036397 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:53.036412 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:53.036441 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:53.036452 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:53.036521 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:53.036716 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 18:36:53.036790 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 18:36:53.038054 28921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:53.025522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:38085 (local address 127.10.131.3:51005)
0620 18:36:53.026653 (+ 1131us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:53.026715 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:53.029434 (+ 2719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:53.030326 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:53.030339 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:53.030484 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:53.031295 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:53.031321 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.032681 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.032686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:53.034508 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:53.034519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.035226 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.035230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:53.035420 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:53.036113 (+ 693us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:36:53.036116 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:53.036158 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:36:53.036233 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:53.036534 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:53.036541 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:53.036547 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 18:36:53.036696 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 18:36:53.037028 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":288,"threads_started":1}
I20260620 18:36:53.039400 28919 heartbeater.cc:347] Connected to a master server at 127.10.131.62:38085
I20260620 18:36:53.039692 28919 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:53.040374 28919 heartbeater.cc:511] Master 127.10.131.62:38085 requested a full tablet report, sending...
I20260620 18:36:53.041702 28471 ts_manager.cc:194] Registered new tserver with Master: 849c0a695b754bd990bc26a00a26cf8b (127.10.131.3:40617)
I20260620 18:36:53.042462 28471 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.131.3:51005
I20260620 18:36:53.046456 10764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:36:53.055760 28660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:53.050438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44070 (local address 127.10.131.62:38085)
0620 18:36:53.050629 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:53.050633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:53.051019 (+ 386us) server_negotiation.cc:407] Connection header received
0620 18:36:53.051150 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:53.051154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:53.051225 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:53.051303 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:53.051756 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.052438 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.053804 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.054206 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.054298 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:53.055234 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:53.055238 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:53.055241 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:53.055256 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:53.055284 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:53.055294 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:53.055358 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:53.055470 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 18:36:53.055541 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 18:36:53.059330 28471 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:36:53.066075 28471 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:36:53.101964 28935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:53.079450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46076 (local address 127.10.131.3:40617)
0620 18:36:53.092811 (+ 13361us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:53.092817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:53.092907 (+ 90us) server_negotiation.cc:407] Connection header received
0620 18:36:53.093005 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:53.093011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:53.093197 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:53.093398 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:53.094565 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.096140 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.098551 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.100360 (+ 1809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.100515 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:53.100794 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:53.101387 (+ 593us) server_negotiation.cc:299] Negotiation successful
0620 18:36:53.101556 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13220,"thread_start_us":13066,"threads_started":1}
I20260620 18:36:53.102169 28933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:53.078405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46794 (local address 127.10.131.1:42159)
0620 18:36:53.094277 (+ 15872us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:53.094284 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:53.094313 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:36:53.094406 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:53.094415 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:53.094670 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:53.095594 (+ 924us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:53.095817 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.097215 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.099729 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.100832 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.101008 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:53.101242 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:53.101636 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 18:36:53.101805 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15700,"thread_start_us":15593,"threads_started":1}
I20260620 18:36:53.102000 28932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:53.078269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:42159 (local address 127.0.0.1:46794)
0620 18:36:53.089771 (+ 11502us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:53.089849 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:53.090140 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:53.094877 (+ 4737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:53.094882 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:53.094905 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:53.095375 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:53.095386 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.097351 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.097356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:53.099566 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:53.099575 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.101047 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.101054 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:53.101238 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:53.101278 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 18:36:53.101371 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 18:36:53.101608 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11078,"mutex_wait_us":31,"thread_start_us":13072,"threads_started":1}
I20260620 18:36:53.102691 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:53.080317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60606 (local address 127.10.131.2:34941)
0620 18:36:53.092344 (+ 12027us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:53.092361 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:53.094204 (+ 1843us) server_negotiation.cc:407] Connection header received
0620 18:36:53.094331 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:53.094337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:53.094537 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:53.094742 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:53.095683 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.096981 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.099549 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.100973 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.101098 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:53.101347 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:53.102116 (+ 769us) server_negotiation.cc:299] Negotiation successful
0620 18:36:53.102266 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11830,"thread_start_us":11704,"threads_started":1}
I20260620 18:36:53.102753 28934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:53.078901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:34941 (local address 127.0.0.1:60606)
0620 18:36:53.094069 (+ 15168us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:53.094088 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:53.094207 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:53.095016 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:53.095025 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:53.095053 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:53.095500 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:53.095512 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.097109 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.097115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:53.099311 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:53.099325 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.101465 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.101471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:53.101665 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:53.101694 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 18:36:53.101774 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 18:36:53.101878 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15029,"thread_start_us":14843,"threads_started":1}
I20260620 18:36:53.103190 28931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:53.078408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:40617 (local address 127.0.0.1:46076)
0620 18:36:53.091601 (+ 13193us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:53.091619 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:53.091724 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:53.093403 (+ 1679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:53.093410 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:53.093463 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:53.094380 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:53.094391 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.096280 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.096285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:53.098258 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:53.098274 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.100930 (+ 2656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.100936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:53.101135 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:53.101192 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 18:36:53.101336 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 18:36:53.102925 (+ 1589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13068,"mutex_wait_us":46,"thread_start_us":11999,"threads_started":1}
I20260620 18:36:53.106509 28854 tablet_service.cc:1467] Processing CreateTablet for tablet 23910f75099a4ebbb0ab8622bf6157ff (DEFAULT_TABLE table=test-table [id=2d20c2bd0454412ebc39caaf78e86ebf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:53.106689 28724 tablet_service.cc:1467] Processing CreateTablet for tablet 23910f75099a4ebbb0ab8622bf6157ff (DEFAULT_TABLE table=test-table [id=2d20c2bd0454412ebc39caaf78e86ebf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:53.106683 28593 tablet_service.cc:1467] Processing CreateTablet for tablet 23910f75099a4ebbb0ab8622bf6157ff (DEFAULT_TABLE table=test-table [id=2d20c2bd0454412ebc39caaf78e86ebf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:53.107844 28854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23910f75099a4ebbb0ab8622bf6157ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:53.107990 28724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23910f75099a4ebbb0ab8622bf6157ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:53.108350 28593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23910f75099a4ebbb0ab8622bf6157ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:53.119484 28937 tablet_bootstrap.cc:492] T 23910f75099a4ebbb0ab8622bf6157ff P 849c0a695b754bd990bc26a00a26cf8b: Bootstrap starting.
I20260620 18:36:53.123253 28937 tablet_bootstrap.cc:654] T 23910f75099a4ebbb0ab8622bf6157ff P 849c0a695b754bd990bc26a00a26cf8b: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:53.124958 28937 log.cc:824] T 23910f75099a4ebbb0ab8622bf6157ff P 849c0a695b754bd990bc26a00a26cf8b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:53.127956 28938 tablet_bootstrap.cc:492] T 23910f75099a4ebbb0ab8622bf6157ff P 8e2b25f838964e0790441b1f7499706a: Bootstrap starting.
I20260620 18:36:53.130805 28939 tablet_bootstrap.cc:492] T 23910f75099a4ebbb0ab8622bf6157ff P 0c4e34f3900e458eaceb040161168c4f: Bootstrap starting.
I20260620 18:36:53.132493 28938 tablet_bootstrap.cc:654] T 23910f75099a4ebbb0ab8622bf6157ff P 8e2b25f838964e0790441b1f7499706a: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:53.133745 28938 log.cc:824] T 23910f75099a4ebbb0ab8622bf6157ff P 8e2b25f838964e0790441b1f7499706a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:53.136394 28939 tablet_bootstrap.cc:654] T 23910f75099a4ebbb0ab8622bf6157ff P 0c4e34f3900e458eaceb040161168c4f: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:53.137861 28939 log.cc:824] T 23910f75099a4ebbb0ab8622bf6157ff P 0c4e34f3900e458eaceb040161168c4f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:53.139343 28938 tablet_bootstrap.cc:492] T 23910f75099a4ebbb0ab8622bf6157ff P 8e2b25f838964e0790441b1f7499706a: No bootstrap required, opened a new log
I20260620 18:36:53.139623 28938 ts_tablet_manager.cc:1397] T 23910f75099a4ebbb0ab8622bf6157ff P 8e2b25f838964e0790441b1f7499706a: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260620 18:36:53.140750 28939 tablet_bootstrap.cc:492] T 23910f75099a4ebbb0ab8622bf6157ff P 0c4e34f3900e458eaceb040161168c4f: No bootstrap required, opened a new log
I20260620 18:36:53.141038 28939 ts_tablet_manager.cc:1397] T 23910f75099a4ebbb0ab8622bf6157ff P 0c4e34f3900e458eaceb040161168c4f: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 18:36:53.141108 28937 tablet_bootstrap.cc:492] T 23910f75099a4ebbb0ab8622bf6157ff P 849c0a695b754bd990bc26a00a26cf8b: No bootstrap required, opened a new log
I20260620 18:36:53.141379 28937 ts_tablet_manager.cc:1397] T 23910f75099a4ebbb0ab8622bf6157ff P 849c0a695b754bd990bc26a00a26cf8b: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.011s
I20260620 18:36:53.145498 28658 heartbeater.cc:503] Master 127.10.131.62:38085 was elected leader, sending a full tablet report...
I20260620 18:36:53.150285 28939 raft_consensus.cc:348] T 23910f75099a4ebbb0ab8622bf6157ff P 0c4e34f3900e458eaceb040161168c4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4e34f3900e458eaceb040161168c4f" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 42159 } } peers { permanent_uuid: "849c0a695b754bd990bc26a00a26cf8b" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40617 } } peers { permanent_uuid: "8e2b25f838964e0790441b1f7499706a" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 34941 } }
I20260620 18:36:53.150322 28938 raft_consensus.cc:348] T 23910f75099a4ebbb0ab8622bf6157ff P 8e2b25f838964e0790441b1f7499706a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4e34f3900e458eaceb040161168c4f" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 42159 } } peers { permanent_uuid: "849c0a695b754bd990bc26a00a26cf8b" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40617 } } peers { permanent_uuid: "8e2b25f838964e0790441b1f7499706a" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 34941 } }
I20260620 18:36:53.150646 28939 raft_consensus.cc:374] T 23910f75099a4ebbb0ab8622bf6157ff P 0c4e34f3900e458eaceb040161168c4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:53.150727 28939 raft_consensus.cc:729] T 23910f75099a4ebbb0ab8622bf6157ff P 0c4e34f3900e458eaceb040161168c4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c4e34f3900e458eaceb040161168c4f, State: Initialized, Role: FOLLOWER
I20260620 18:36:53.150744 28938 raft_consensus.cc:374] T 23910f75099a4ebbb0ab8622bf6157ff P 8e2b25f838964e0790441b1f7499706a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:53.150851 28938 raft_consensus.cc:729] T 23910f75099a4ebbb0ab8622bf6157ff P 8e2b25f838964e0790441b1f7499706a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e2b25f838964e0790441b1f7499706a, State: Initialized, Role: FOLLOWER
I20260620 18:36:53.151363 28939 consensus_queue.cc:260] T 23910f75099a4ebbb0ab8622bf6157ff P 0c4e34f3900e458eaceb040161168c4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4e34f3900e458eaceb040161168c4f" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 42159 } } peers { permanent_uuid: "849c0a695b754bd990bc26a00a26cf8b" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40617 } } peers { permanent_uuid: "8e2b25f838964e0790441b1f7499706a" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 34941 } }
I20260620 18:36:53.151486 28938 consensus_queue.cc:260] T 23910f75099a4ebbb0ab8622bf6157ff P 8e2b25f838964e0790441b1f7499706a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4e34f3900e458eaceb040161168c4f" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 42159 } } peers { permanent_uuid: "849c0a695b754bd990bc26a00a26cf8b" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40617 } } peers { permanent_uuid: "8e2b25f838964e0790441b1f7499706a" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 34941 } }
I20260620 18:36:53.152716 28789 heartbeater.cc:503] Master 127.10.131.62:38085 was elected leader, sending a full tablet report...
I20260620 18:36:53.153411 28938 ts_tablet_manager.cc:1428] T 23910f75099a4ebbb0ab8622bf6157ff P 8e2b25f838964e0790441b1f7499706a: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260620 18:36:53.152463 28937 raft_consensus.cc:348] T 23910f75099a4ebbb0ab8622bf6157ff P 849c0a695b754bd990bc26a00a26cf8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4e34f3900e458eaceb040161168c4f" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 42159 } } peers { permanent_uuid: "849c0a695b754bd990bc26a00a26cf8b" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40617 } } peers { permanent_uuid: "8e2b25f838964e0790441b1f7499706a" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 34941 } }
I20260620 18:36:53.154327 28937 raft_consensus.cc:374] T 23910f75099a4ebbb0ab8622bf6157ff P 849c0a695b754bd990bc26a00a26cf8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:53.154438 28937 raft_consensus.cc:729] T 23910f75099a4ebbb0ab8622bf6157ff P 849c0a695b754bd990bc26a00a26cf8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 849c0a695b754bd990bc26a00a26cf8b, State: Initialized, Role: FOLLOWER
I20260620 18:36:53.155037 28937 consensus_queue.cc:260] T 23910f75099a4ebbb0ab8622bf6157ff P 849c0a695b754bd990bc26a00a26cf8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4e34f3900e458eaceb040161168c4f" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 42159 } } peers { permanent_uuid: "849c0a695b754bd990bc26a00a26cf8b" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40617 } } peers { permanent_uuid: "8e2b25f838964e0790441b1f7499706a" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 34941 } }
I20260620 18:36:53.156142 28919 heartbeater.cc:503] Master 127.10.131.62:38085 was elected leader, sending a full tablet report...
I20260620 18:36:53.156879 28937 ts_tablet_manager.cc:1428] T 23910f75099a4ebbb0ab8622bf6157ff P 849c0a695b754bd990bc26a00a26cf8b: Time spent starting tablet: real 0.015s user 0.010s sys 0.003s
I20260620 18:36:53.159366 28939 ts_tablet_manager.cc:1428] T 23910f75099a4ebbb0ab8622bf6157ff P 0c4e34f3900e458eaceb040161168c4f: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
I20260620 18:36:53.172935 28945 raft_consensus.cc:482] T 23910f75099a4ebbb0ab8622bf6157ff P 849c0a695b754bd990bc26a00a26cf8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:36:53.173223 28945 raft_consensus.cc:504] T 23910f75099a4ebbb0ab8622bf6157ff P 849c0a695b754bd990bc26a00a26cf8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4e34f3900e458eaceb040161168c4f" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 42159 } } peers { permanent_uuid: "849c0a695b754bd990bc26a00a26cf8b" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40617 } } peers { permanent_uuid: "8e2b25f838964e0790441b1f7499706a" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 34941 } }
I20260620 18:36:53.174993 28945 leader_election.cc:288] T 23910f75099a4ebbb0ab8622bf6157ff P 849c0a695b754bd990bc26a00a26cf8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c4e34f3900e458eaceb040161168c4f (127.10.131.1:42159), 8e2b25f838964e0790441b1f7499706a (127.10.131.2:34941)
I20260620 18:36:53.181149 28921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:53.175501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:42159 (local address 127.10.131.3:48471)
0620 18:36:53.175685 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:53.175701 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:53.175868 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:53.176239 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:53.176245 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:53.176269 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:53.176453 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:53.176464 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.177606 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.177612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:53.179452 (+ 1840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:53.179468 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.180519 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.180527 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:53.180698 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:53.180720 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:36:53.180785 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 18:36:53.180938 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 18:36:53.181241 28933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:53.175297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:48471 (local address 127.10.131.1:42159)
0620 18:36:53.175489 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:53.175494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:53.175777 (+ 283us) server_negotiation.cc:407] Connection header received
0620 18:36:53.176015 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:53.176020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:53.176100 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:53.176196 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:53.176607 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.177493 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.179618 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.180376 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.180510 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:53.180557 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:53.180860 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 18:36:53.180997 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:36:53.182126 28613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23910f75099a4ebbb0ab8622bf6157ff" candidate_uuid: "849c0a695b754bd990bc26a00a26cf8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c4e34f3900e458eaceb040161168c4f" is_pre_election: true
I20260620 18:36:53.182610 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:53.175526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:58483 (local address 127.10.131.2:34941)
0620 18:36:53.175746 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:53.175751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:53.177030 (+ 1279us) server_negotiation.cc:407] Connection header received
0620 18:36:53.177212 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:53.177216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:53.177269 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:53.177363 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:53.177939 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.178831 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.180665 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.181762 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.181939 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:53.182006 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:53.182246 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 18:36:53.182366 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:36:53.182667 28946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:53.176161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:34941 (local address 127.10.131.3:58483)
0620 18:36:53.176937 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:53.176950 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:53.177079 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:53.177373 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:53.177377 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:53.177398 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:53.177774 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:53.177783 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.178983 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.178988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:53.180486 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:53.180497 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.181925 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.181931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:53.182098 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:53.182120 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:36:53.182220 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 18:36:53.182338 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":443,"threads_started":1}
I20260620 18:36:53.182632 28613 raft_consensus.cc:2454] T 23910f75099a4ebbb0ab8622bf6157ff P 0c4e34f3900e458eaceb040161168c4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 849c0a695b754bd990bc26a00a26cf8b in term 0.
I20260620 18:36:53.183350 28807 leader_election.cc:302] T 23910f75099a4ebbb0ab8622bf6157ff P 849c0a695b754bd990bc26a00a26cf8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4e34f3900e458eaceb040161168c4f, 849c0a695b754bd990bc26a00a26cf8b; no voters:
I20260620 18:36:53.183691 28945 raft_consensus.cc:2781] T 23910f75099a4ebbb0ab8622bf6157ff P 849c0a695b754bd990bc26a00a26cf8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:36:53.183809 28945 raft_consensus.cc:482] T 23910f75099a4ebbb0ab8622bf6157ff P 849c0a695b754bd990bc26a00a26cf8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:36:53.183938 28945 raft_consensus.cc:3037] T 23910f75099a4ebbb0ab8622bf6157ff P 849c0a695b754bd990bc26a00a26cf8b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:53.183724 28744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23910f75099a4ebbb0ab8622bf6157ff" candidate_uuid: "849c0a695b754bd990bc26a00a26cf8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e2b25f838964e0790441b1f7499706a" is_pre_election: true
I20260620 18:36:53.184244 28744 raft_consensus.cc:2454] T 23910f75099a4ebbb0ab8622bf6157ff P 8e2b25f838964e0790441b1f7499706a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 849c0a695b754bd990bc26a00a26cf8b in term 0.
I20260620 18:36:53.186246 28945 raft_consensus.cc:504] T 23910f75099a4ebbb0ab8622bf6157ff P 849c0a695b754bd990bc26a00a26cf8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4e34f3900e458eaceb040161168c4f" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 42159 } } peers { permanent_uuid: "849c0a695b754bd990bc26a00a26cf8b" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40617 } } peers { permanent_uuid: "8e2b25f838964e0790441b1f7499706a" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 34941 } }
I20260620 18:36:53.186832 28945 leader_election.cc:288] T 23910f75099a4ebbb0ab8622bf6157ff P 849c0a695b754bd990bc26a00a26cf8b [CANDIDATE]: Term 1 election: Requested vote from peers 0c4e34f3900e458eaceb040161168c4f (127.10.131.1:42159), 8e2b25f838964e0790441b1f7499706a (127.10.131.2:34941)
I20260620 18:36:53.187178 28613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23910f75099a4ebbb0ab8622bf6157ff" candidate_uuid: "849c0a695b754bd990bc26a00a26cf8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c4e34f3900e458eaceb040161168c4f"
I20260620 18:36:53.187283 28744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23910f75099a4ebbb0ab8622bf6157ff" candidate_uuid: "849c0a695b754bd990bc26a00a26cf8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e2b25f838964e0790441b1f7499706a"
I20260620 18:36:53.187377 28613 raft_consensus.cc:3037] T 23910f75099a4ebbb0ab8622bf6157ff P 0c4e34f3900e458eaceb040161168c4f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:53.187481 28744 raft_consensus.cc:3037] T 23910f75099a4ebbb0ab8622bf6157ff P 8e2b25f838964e0790441b1f7499706a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:53.189618 28613 raft_consensus.cc:2454] T 23910f75099a4ebbb0ab8622bf6157ff P 0c4e34f3900e458eaceb040161168c4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 849c0a695b754bd990bc26a00a26cf8b in term 1.
I20260620 18:36:53.189710 28744 raft_consensus.cc:2454] T 23910f75099a4ebbb0ab8622bf6157ff P 8e2b25f838964e0790441b1f7499706a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 849c0a695b754bd990bc26a00a26cf8b in term 1.
I20260620 18:36:53.190071 28807 leader_election.cc:302] T 23910f75099a4ebbb0ab8622bf6157ff P 849c0a695b754bd990bc26a00a26cf8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4e34f3900e458eaceb040161168c4f, 849c0a695b754bd990bc26a00a26cf8b; no voters:
I20260620 18:36:53.190321 28945 raft_consensus.cc:2781] T 23910f75099a4ebbb0ab8622bf6157ff P 849c0a695b754bd990bc26a00a26cf8b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:36:53.190727 28945 raft_consensus.cc:686] T 23910f75099a4ebbb0ab8622bf6157ff P 849c0a695b754bd990bc26a00a26cf8b [term 1 LEADER]: Becoming Leader. State: Replica: 849c0a695b754bd990bc26a00a26cf8b, State: Running, Role: LEADER
I20260620 18:36:53.191164 28945 consensus_queue.cc:237] T 23910f75099a4ebbb0ab8622bf6157ff P 849c0a695b754bd990bc26a00a26cf8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4e34f3900e458eaceb040161168c4f" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 42159 } } peers { permanent_uuid: "849c0a695b754bd990bc26a00a26cf8b" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40617 } } peers { permanent_uuid: "8e2b25f838964e0790441b1f7499706a" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 34941 } }
I20260620 18:36:53.195804 28471 catalog_manager.cc:5495] T 23910f75099a4ebbb0ab8622bf6157ff P 849c0a695b754bd990bc26a00a26cf8b reported cstate change: term changed from 0 to 1, leader changed from <none> to 849c0a695b754bd990bc26a00a26cf8b (127.10.131.3). New cstate: current_term: 1 leader_uuid: "849c0a695b754bd990bc26a00a26cf8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4e34f3900e458eaceb040161168c4f" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 42159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "849c0a695b754bd990bc26a00a26cf8b" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e2b25f838964e0790441b1f7499706a" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 34941 } health_report { overall_health: UNKNOWN } } }
I20260620 18:36:53.233991 28935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:53.228266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46086 (local address 127.10.131.3:40617)
0620 18:36:53.228491 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:53.228497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:53.228567 (+ 70us) server_negotiation.cc:407] Connection header received
0620 18:36:53.228889 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:53.228893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:53.228968 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:53.229149 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:36:53.229510 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.230300 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.231876 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.232285 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.232420 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:53.232936 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:36:53.233407 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:36:53.233580 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:53.233666 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 18:36:53.233742 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 18:36:53.242508 28613 raft_consensus.cc:1261] T 23910f75099a4ebbb0ab8622bf6157ff P 0c4e34f3900e458eaceb040161168c4f [term 1 FOLLOWER]: Refusing update from remote peer 849c0a695b754bd990bc26a00a26cf8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:36:53.242508 28744 raft_consensus.cc:1261] T 23910f75099a4ebbb0ab8622bf6157ff P 8e2b25f838964e0790441b1f7499706a [term 1 FOLLOWER]: Refusing update from remote peer 849c0a695b754bd990bc26a00a26cf8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:36:53.243460 28947 consensus_queue.cc:1035] T 23910f75099a4ebbb0ab8622bf6157ff P 849c0a695b754bd990bc26a00a26cf8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4e34f3900e458eaceb040161168c4f" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 42159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:36:53.243796 28945 consensus_queue.cc:1035] T 23910f75099a4ebbb0ab8622bf6157ff P 849c0a695b754bd990bc26a00a26cf8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e2b25f838964e0790441b1f7499706a" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 34941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:36:53.260541 28952 mvcc.cc:205] Tried to move back new op lower bound from 7298992591831437312 to 7298992591634948096. Current Snapshot: MvccSnapshot[applied={T|T < 7298992591831437312}]
I20260620 18:36:53.269556 28470 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:36:53.269850 28470 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:36:53.273368 28470 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eaf0d22b669641dd9e40b5f45a3603b4: Sending DeleteTablet for 3 replicas of tablet 23910f75099a4ebbb0ab8622bf6157ff
I20260620 18:36:53.274531 28593 tablet_service.cc:1514] Processing DeleteTablet for tablet 23910f75099a4ebbb0ab8622bf6157ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:53 UTC) from {username='slave'} at 127.0.0.1:46794
I20260620 18:36:53.274668 28724 tablet_service.cc:1514] Processing DeleteTablet for tablet 23910f75099a4ebbb0ab8622bf6157ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:53 UTC) from {username='slave'} at 127.0.0.1:60606
I20260620 18:36:53.275486 28854 tablet_service.cc:1514] Processing DeleteTablet for tablet 23910f75099a4ebbb0ab8622bf6157ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:53 UTC) from {username='slave'} at 127.0.0.1:46076
I20260620 18:36:53.276403 28961 tablet_replica.cc:330] T 23910f75099a4ebbb0ab8622bf6157ff P 849c0a695b754bd990bc26a00a26cf8b: stopping tablet replica
I20260620 18:36:53.276968 28961 raft_consensus.cc:2229] T 23910f75099a4ebbb0ab8622bf6157ff P 849c0a695b754bd990bc26a00a26cf8b [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:36:53.277659 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 28531
I20260620 18:36:53.277851 28961 raft_consensus.cc:2258] T 23910f75099a4ebbb0ab8622bf6157ff P 849c0a695b754bd990bc26a00a26cf8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:53.280666 28961 ts_tablet_manager.cc:1905] T 23910f75099a4ebbb0ab8622bf6157ff P 849c0a695b754bd990bc26a00a26cf8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:53.285372 28959 tablet_replica.cc:330] T 23910f75099a4ebbb0ab8622bf6157ff P 0c4e34f3900e458eaceb040161168c4f: stopping tablet replica
I20260620 18:36:53.285804 28959 raft_consensus.cc:2229] T 23910f75099a4ebbb0ab8622bf6157ff P 0c4e34f3900e458eaceb040161168c4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:36:53.286334 28959 raft_consensus.cc:2258] T 23910f75099a4ebbb0ab8622bf6157ff P 0c4e34f3900e458eaceb040161168c4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:53.287173 28960 tablet_replica.cc:330] T 23910f75099a4ebbb0ab8622bf6157ff P 8e2b25f838964e0790441b1f7499706a: stopping tablet replica
I20260620 18:36:53.287586 28960 raft_consensus.cc:2229] T 23910f75099a4ebbb0ab8622bf6157ff P 8e2b25f838964e0790441b1f7499706a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:36:53.288074 28960 raft_consensus.cc:2258] T 23910f75099a4ebbb0ab8622bf6157ff P 8e2b25f838964e0790441b1f7499706a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:53.289980 28961 ts_tablet_manager.cc:1918] T 23910f75099a4ebbb0ab8622bf6157ff P 849c0a695b754bd990bc26a00a26cf8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:36:53.290205 28961 log.cc:1196] T 23910f75099a4ebbb0ab8622bf6157ff P 849c0a695b754bd990bc26a00a26cf8b: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-2/wal/wals/23910f75099a4ebbb0ab8622bf6157ff
I20260620 18:36:53.290300 28960 ts_tablet_manager.cc:1905] T 23910f75099a4ebbb0ab8622bf6157ff P 8e2b25f838964e0790441b1f7499706a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:53.290293 28959 ts_tablet_manager.cc:1905] T 23910f75099a4ebbb0ab8622bf6157ff P 0c4e34f3900e458eaceb040161168c4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:53.290915 28961 ts_tablet_manager.cc:1939] T 23910f75099a4ebbb0ab8622bf6157ff P 849c0a695b754bd990bc26a00a26cf8b: Deleting consensus metadata
I20260620 18:36:53.292737 28456 catalog_manager.cc:4841] TS 849c0a695b754bd990bc26a00a26cf8b (127.10.131.3:40617): tablet 23910f75099a4ebbb0ab8622bf6157ff (table test-table [id=2d20c2bd0454412ebc39caaf78e86ebf]) successfully deleted
I20260620 18:36:53.295945 28960 ts_tablet_manager.cc:1918] T 23910f75099a4ebbb0ab8622bf6157ff P 8e2b25f838964e0790441b1f7499706a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:36:53.296118 28960 log.cc:1196] T 23910f75099a4ebbb0ab8622bf6157ff P 8e2b25f838964e0790441b1f7499706a: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-1/wal/wals/23910f75099a4ebbb0ab8622bf6157ff
I20260620 18:36:53.296625 28960 ts_tablet_manager.cc:1939] T 23910f75099a4ebbb0ab8622bf6157ff P 8e2b25f838964e0790441b1f7499706a: Deleting consensus metadata
I20260620 18:36:53.298401 28457 catalog_manager.cc:4841] TS 8e2b25f838964e0790441b1f7499706a (127.10.131.2:34941): tablet 23910f75099a4ebbb0ab8622bf6157ff (table test-table [id=2d20c2bd0454412ebc39caaf78e86ebf]) successfully deleted
I20260620 18:36:53.300124 28959 ts_tablet_manager.cc:1918] T 23910f75099a4ebbb0ab8622bf6157ff P 0c4e34f3900e458eaceb040161168c4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:36:53.300302 28959 log.cc:1196] T 23910f75099a4ebbb0ab8622bf6157ff P 0c4e34f3900e458eaceb040161168c4f: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781980314095189-10764-0/minicluster-data/ts-0/wal/wals/23910f75099a4ebbb0ab8622bf6157ff
I20260620 18:36:53.300732 28959 ts_tablet_manager.cc:1939] T 23910f75099a4ebbb0ab8622bf6157ff P 0c4e34f3900e458eaceb040161168c4f: Deleting consensus metadata
I20260620 18:36:53.302441 28455 catalog_manager.cc:4841] TS 0c4e34f3900e458eaceb040161168c4f (127.10.131.1:42159): tablet 23910f75099a4ebbb0ab8622bf6157ff (table test-table [id=2d20c2bd0454412ebc39caaf78e86ebf]) successfully deleted
I20260620 18:36:53.304258 28933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:53.278664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46796 (local address 127.10.131.1:42159)
0620 18:36:53.278888 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:53.278897 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:53.296271 (+ 17374us) server_negotiation.cc:407] Connection header received
0620 18:36:53.296781 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:53.296786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:53.296916 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:53.296996 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:53.297637 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.298875 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.300826 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.301557 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.301692 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:53.302950 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:53.302958 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:53.302971 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:53.303506 (+ 535us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:53.303548 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:53.303567 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:53.303662 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:53.303819 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 18:36:53.303901 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:36:53.304982 28653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:53.394376 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 28531
I20260620 18:36:53.411954 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 28662
I20260620 18:36:53.419025 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:53.412797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60622 (local address 127.10.131.2:34941)
0620 18:36:53.412990 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:53.412996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:53.413042 (+ 46us) server_negotiation.cc:407] Connection header received
0620 18:36:53.413269 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:53.413275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:53.413386 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:53.413498 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:53.413909 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.414619 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.415908 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.416326 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.416459 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:53.417626 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:53.417635 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:53.417668 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:53.418262 (+ 594us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:53.418320 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:53.418338 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:53.418455 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:53.418585 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 18:36:53.418697 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 18:36:53.419654 28784 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:53.530131 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 28662
I20260620 18:36:53.546661 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 28792
I20260620 18:36:53.554855 28935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:53.547427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46094 (local address 127.10.131.3:40617)
0620 18:36:53.547642 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:53.547648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:53.547682 (+ 34us) server_negotiation.cc:407] Connection header received
0620 18:36:53.547833 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:53.547839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:53.547936 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:53.548087 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:53.548550 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.549858 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.551159 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:53.551822 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:53.551951 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:53.553148 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:53.553159 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:53.553178 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:53.554020 (+ 842us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:53.554100 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:53.554132 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:53.554249 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:53.554361 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 18:36:53.554462 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 18:36:53.555615 28914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:53.679874 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 28792
I20260620 18:36:53.699174 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 28440
I20260620 18:36:53.700171 28501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:53.806265 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 28440
2026-06-20T18:36:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2735 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:36:53 dist-test-slave-cbn3 krb5kdc[28969](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:36:53 dist-test-slave-cbn3 krb5kdc[28969](info): set up 2 sockets
Jun 20 18:36:53 dist-test-slave-cbn3 krb5kdc[28969](info): commencing operation
krb5kdc: starting...
W20260620 18:36:54.904990 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:36:55 dist-test-slave-cbn3 krb5kdc[28969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:36:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:36:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.131.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
I20260620 18:36:55.151685 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:33639
--webserver_interface=127.10.131.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:43747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:33639
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:55.483111 28982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:55.483425 28982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:55.483619 28982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:55.491016 28982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:36:55.491144 28982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:55.491267 28982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:55.491374 28982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:36:55.491463 28982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:36:55.501585 28982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:43747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:33639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:33639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:36:55.502983 28982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:55.504326 28982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:55.516676 28990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:55.516597 28988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:55.517114 28987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:55.518183 28982 server_base.cc:1034] running on GCE node
I20260620 18:36:55.518985 28982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:55.520480 28982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:55.521711 28982 hybrid_clock.cc:648] HybridClock initialized: now 1781980615521669 us; error 54 us; skew 500 ppm
I20260620 18:36:55.522079 28982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:55 dist-test-slave-cbn3 krb5kdc[28969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980615, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:55.525264 28982 init.cc:377] Logged in from keytab as kudu/127.10.131.62@KRBTEST.COM (short username kudu)
I20260620 18:36:55.526865 28982 webserver.cc:460] Webserver started at http://127.10.131.62:42483/ using document root <none> and password file <none>
I20260620 18:36:55.527575 28982 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:55.527722 28982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:55.528035 28982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:55.531137 28982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "e4745180ec054466936fe98e736c8532"
format_stamp: "Formatted at 2026-06-20 18:36:55 on dist-test-slave-cbn3"
server_key: "972c85d2aff9eeaf95631de18f5840d5"
server_key_iv: "1466267b6221f1aea8249b8d133fed46"
server_key_version: "encryptionkey@0"
I20260620 18:36:55.531908 28982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "e4745180ec054466936fe98e736c8532"
format_stamp: "Formatted at 2026-06-20 18:36:55 on dist-test-slave-cbn3"
server_key: "972c85d2aff9eeaf95631de18f5840d5"
server_key_iv: "1466267b6221f1aea8249b8d133fed46"
server_key_version: "encryptionkey@0"
I20260620 18:36:55.537276 28982 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:36:55.540795 28997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:55.542163 28982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:36:55.542366 28982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "e4745180ec054466936fe98e736c8532"
format_stamp: "Formatted at 2026-06-20 18:36:55 on dist-test-slave-cbn3"
server_key: "972c85d2aff9eeaf95631de18f5840d5"
server_key_iv: "1466267b6221f1aea8249b8d133fed46"
server_key_version: "encryptionkey@0"
I20260620 18:36:55.542584 28982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:55.561975 28982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:55.566047 28982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:55.566346 28982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:55.585093 28982 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:33639
I20260620 18:36:55.585119 29048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:33639 every 8 connection(s)
I20260620 18:36:55.586486 28982 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:36:55.590644 29049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:55.595739 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 28982
I20260620 18:36:55.595952 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:36:55.596464 10764 external_mini_cluster.cc:1428] Setting key bd06aff885d3c485bf4937cba5726aff
I20260620 18:36:55.602007 29049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4745180ec054466936fe98e736c8532: Bootstrap starting.
I20260620 18:36:55.605948 29049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4745180ec054466936fe98e736c8532: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:55.607234 29049 log.cc:824] T 00000000000000000000000000000000 P e4745180ec054466936fe98e736c8532: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:55.611636 29049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4745180ec054466936fe98e736c8532: No bootstrap required, opened a new log
Jun 20 18:36:55 dist-test-slave-cbn3 krb5kdc[28969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:36:55.621821 29049 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4745180ec054466936fe98e736c8532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4745180ec054466936fe98e736c8532" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 33639 } }
I20260620 18:36:55.622241 29049 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4745180ec054466936fe98e736c8532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:55.622354 29049 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4745180ec054466936fe98e736c8532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4745180ec054466936fe98e736c8532, State: Initialized, Role: FOLLOWER
I20260620 18:36:55.622666 29052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:55.598319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54238 (local address 127.10.131.62:33639)
0620 18:36:55.599179 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:55.599190 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:55.599235 (+ 45us) server_negotiation.cc:407] Connection header received
0620 18:36:55.601071 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:55.601094 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:55.601496 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:55.601811 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:55.603601 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:55.604998 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:55.606891 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:55.607714 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:55.607911 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:55.614088 (+ 6177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:55.614104 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:55.614125 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:55.614181 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:55.618134 (+ 3953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:55.618913 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:55.618923 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:55.618932 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:55.619115 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:55.619524 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:55.619530 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:55.619534 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:55.620130 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:55.620464 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:55.620959 (+ 495us) server_negotiation.cc:299] Negotiation successful
0620 18:36:55.621419 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":402,"threads_started":1}
I20260620 18:36:55.623118 29049 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4745180ec054466936fe98e736c8532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4745180ec054466936fe98e736c8532" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 33639 } }
I20260620 18:36:55.623432 29049 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4745180ec054466936fe98e736c8532 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:36:55.623581 29049 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4745180ec054466936fe98e736c8532 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:36:55.623759 29049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4745180ec054466936fe98e736c8532 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:55.626492 29049 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4745180ec054466936fe98e736c8532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4745180ec054466936fe98e736c8532" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 33639 } }
I20260620 18:36:55.626947 29049 leader_election.cc:302] T 00000000000000000000000000000000 P e4745180ec054466936fe98e736c8532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4745180ec054466936fe98e736c8532; no voters:
I20260620 18:36:55.627454 29049 leader_election.cc:288] T 00000000000000000000000000000000 P e4745180ec054466936fe98e736c8532 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:36:55.627597 29054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4745180ec054466936fe98e736c8532 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:36:55.628255 29054 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4745180ec054466936fe98e736c8532 [term 1 LEADER]: Becoming Leader. State: Replica: e4745180ec054466936fe98e736c8532, State: Running, Role: LEADER
I20260620 18:36:55.628749 29054 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4745180ec054466936fe98e736c8532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4745180ec054466936fe98e736c8532" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 33639 } }
I20260620 18:36:55.629601 29049 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4745180ec054466936fe98e736c8532 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:36:55.631696 29056 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4745180ec054466936fe98e736c8532 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4745180ec054466936fe98e736c8532. Latest consensus state: current_term: 1 leader_uuid: "e4745180ec054466936fe98e736c8532" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4745180ec054466936fe98e736c8532" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 33639 } } }
I20260620 18:36:55.632144 29056 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4745180ec054466936fe98e736c8532 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:55.633045 29055 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4745180ec054466936fe98e736c8532 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4745180ec054466936fe98e736c8532" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4745180ec054466936fe98e736c8532" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 33639 } } }
I20260620 18:36:55.633394 29055 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4745180ec054466936fe98e736c8532 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:55.635318 29059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:36:55.641742 29059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:36:55.654412 29059 catalog_manager.cc:1347] Generated new cluster ID: 7c19dd3349c84c16b3890455bf1a2786
I20260620 18:36:55.654557 29059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:36:55.668263 29059 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:36:55.669610 29059 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:36:55.675963 29059 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4745180ec054466936fe98e736c8532: Generated new TSK 0
I20260620 18:36:55.676753 29059 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.131.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab.
Entry for principal kudu/127.10.131.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab.
I20260620 18:36:55.739795 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.1:0
--local_ip_for_outbound_sockets=127.10.131.1
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:33639
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:43747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:56.067916 29075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:56.068240 29075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:56.068359 29075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:56.075932 29075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:56.076048 29075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:56.076252 29075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.1
I20260620 18:36:56.086226 29075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:43747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:33639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.1
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:36:56.087477 29075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:56.088779 29075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:56.101688 29080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:56.101617 29081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:56.101665 29083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:56.102442 29075 server_base.cc:1034] running on GCE node
I20260620 18:36:56.103081 29075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:56.103972 29075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:56.105197 29075 hybrid_clock.cc:648] HybridClock initialized: now 1781980616105150 us; error 54 us; skew 500 ppm
I20260620 18:36:56.105518 29075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:56 dist-test-slave-cbn3 krb5kdc[28969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980616, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:56.108636 29075 init.cc:377] Logged in from keytab as kudu/127.10.131.1@KRBTEST.COM (short username kudu)
I20260620 18:36:56.110105 29075 webserver.cc:460] Webserver started at http://127.10.131.1:36463/ using document root <none> and password file <none>
I20260620 18:36:56.110771 29075 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:56.110922 29075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:56.111234 29075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:56.114351 29075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-0/data/instance:
uuid: "f889374377284726bca487bd8bdc6ccd"
format_stamp: "Formatted at 2026-06-20 18:36:56 on dist-test-slave-cbn3"
server_key: "cef472db6c470b625e98d4358e630ccd"
server_key_iv: "bcbb3fce4d80d0bb1ff4d54b39602d5a"
server_key_version: "encryptionkey@0"
I20260620 18:36:56.115099 29075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance:
uuid: "f889374377284726bca487bd8bdc6ccd"
format_stamp: "Formatted at 2026-06-20 18:36:56 on dist-test-slave-cbn3"
server_key: "cef472db6c470b625e98d4358e630ccd"
server_key_iv: "bcbb3fce4d80d0bb1ff4d54b39602d5a"
server_key_version: "encryptionkey@0"
I20260620 18:36:56.120106 29075 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260620 18:36:56.123543 29090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:56.124791 29075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 18:36:56.124992 29075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-0/wal
uuid: "f889374377284726bca487bd8bdc6ccd"
format_stamp: "Formatted at 2026-06-20 18:36:56 on dist-test-slave-cbn3"
server_key: "cef472db6c470b625e98d4358e630ccd"
server_key_iv: "bcbb3fce4d80d0bb1ff4d54b39602d5a"
server_key_version: "encryptionkey@0"
I20260620 18:36:56.125201 29075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:56.140048 29075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:56.144181 29075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:56.144472 29075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:56.145433 29075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:56.146785 29075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:56.146883 29075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:56.147013 29075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:56.147091 29075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:56.178764 29075 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.1:43403
I20260620 18:36:56.178808 29202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.1:43403 every 8 connection(s)
I20260620 18:36:56.180001 29075 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
I20260620 18:36:56.183270 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 29075
I20260620 18:36:56.183466 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance
I20260620 18:36:56.183938 10764 external_mini_cluster.cc:1428] Setting key e4de58f1466d214874b2fe1fa44926e7
Jun 20 18:36:56 dist-test-slave-cbn3 krb5kdc[28969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980616, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:36:56.202431 29205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:56.182478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:54219 (local address 127.10.131.62:33639)
0620 18:36:56.182880 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:56.182885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:56.183967 (+ 1082us) server_negotiation.cc:407] Connection header received
0620 18:36:56.186138 (+ 2171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:56.186144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:56.186230 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:56.186357 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:56.187774 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:56.189027 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:56.190591 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:56.191357 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:56.191504 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:56.195312 (+ 3808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:56.195322 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:56.195330 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:56.195386 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:56.198972 (+ 3586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:56.199762 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:56.199770 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:56.199777 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:56.199926 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:56.200329 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:56.200344 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:56.200351 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:56.200729 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:56.200903 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:56.201651 (+ 748us) server_negotiation.cc:299] Negotiation successful
0620 18:36:56.201930 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":203,"threads_started":1}
I20260620 18:36:56.203269 29206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:56.182998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:33639 (local address 127.10.131.1:54219)
0620 18:36:56.183772 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:56.183845 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:56.185889 (+ 2044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:56.186550 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:56.186558 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:56.187057 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:56.187600 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:56.187616 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:56.189193 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:56.189197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:56.190435 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:56.190448 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:56.191586 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:56.191591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:56.191798 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:56.192539 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:56.192563 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:56.195066 (+ 2503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:56.199191 (+ 4125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:56.199198 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:56.199212 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:56.199635 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:56.200033 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:56.200037 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:56.200039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:56.200222 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:56.200900 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:56.200916 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:56.201257 (+ 341us) client_negotiation.cc:769] Sending connection context
0620 18:36:56.201502 (+ 245us) client_negotiation.cc:240] Negotiation successful
0620 18:36:56.202018 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":336,"threads_started":1}
I20260620 18:36:56.205035 29203 heartbeater.cc:347] Connected to a master server at 127.10.131.62:33639
I20260620 18:36:56.205408 29203 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:56.206349 29203 heartbeater.cc:511] Master 127.10.131.62:33639 requested a full tablet report, sending...
I20260620 18:36:56.209039 29014 ts_manager.cc:194] Registered new tserver with Master: f889374377284726bca487bd8bdc6ccd (127.10.131.1:43403)
WARNING: no policy specified for kudu/127.10.131.2@KRBTEST.COM; defaulting to no policy
I20260620 18:36:56.211268 29014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.1@KRBTEST.COM'} at 127.10.131.1:54219
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab.
Entry for principal kudu/127.10.131.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab.
I20260620 18:36:56.241531 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.2:0
--local_ip_for_outbound_sockets=127.10.131.2
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:33639
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:43747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:56.559142 29209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:56.559425 29209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:56.559551 29209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:56.566767 29209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:56.566897 29209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:56.567055 29209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.2
I20260620 18:36:56.576638 29209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:43747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:33639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.2
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:36:56.577883 29209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:56.579123 29209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:56.594041 29214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:56.594030 29217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:56.594455 29215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:56.594545 29209 server_base.cc:1034] running on GCE node
I20260620 18:36:56.595568 29209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:56.596626 29209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:56.597852 29209 hybrid_clock.cc:648] HybridClock initialized: now 1781980616597769 us; error 72 us; skew 500 ppm
I20260620 18:36:56.598218 29209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:56 dist-test-slave-cbn3 krb5kdc[28969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980616, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:56.601830 29209 init.cc:377] Logged in from keytab as kudu/127.10.131.2@KRBTEST.COM (short username kudu)
I20260620 18:36:56.603257 29209 webserver.cc:460] Webserver started at http://127.10.131.2:36827/ using document root <none> and password file <none>
I20260620 18:36:56.603983 29209 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:56.604122 29209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:56.604434 29209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:56.607228 29209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-1/data/instance:
uuid: "95918c3a2807477ca6d9d16ec93b94ae"
format_stamp: "Formatted at 2026-06-20 18:36:56 on dist-test-slave-cbn3"
server_key: "701ff44b09a7393d2fc8ee7a8e487553"
server_key_iv: "d87ca7de7166422bbfae704393ba15f8"
server_key_version: "encryptionkey@0"
I20260620 18:36:56.608037 29209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance:
uuid: "95918c3a2807477ca6d9d16ec93b94ae"
format_stamp: "Formatted at 2026-06-20 18:36:56 on dist-test-slave-cbn3"
server_key: "701ff44b09a7393d2fc8ee7a8e487553"
server_key_iv: "d87ca7de7166422bbfae704393ba15f8"
server_key_version: "encryptionkey@0"
I20260620 18:36:56.613147 29209 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 18:36:56.616452 29224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:56.617746 29209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:36:56.617902 29209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-1/wal
uuid: "95918c3a2807477ca6d9d16ec93b94ae"
format_stamp: "Formatted at 2026-06-20 18:36:56 on dist-test-slave-cbn3"
server_key: "701ff44b09a7393d2fc8ee7a8e487553"
server_key_iv: "d87ca7de7166422bbfae704393ba15f8"
server_key_version: "encryptionkey@0"
I20260620 18:36:56.618119 29209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:56.662220 29209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:56.666509 29209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:56.666790 29209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:56.667742 29209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:56.669039 29209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:56.669111 29209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:56.669274 29209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:56.669334 29209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:56.701359 29209 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.2:36943
I20260620 18:36:56.701414 29336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.2:36943 every 8 connection(s)
I20260620 18:36:56.702647 29209 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
I20260620 18:36:56.705312 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 29209
I20260620 18:36:56.705502 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance
I20260620 18:36:56.706054 10764 external_mini_cluster.cc:1428] Setting key 5a35de61238d131705e2c450a4625f79
Jun 20 18:36:56 dist-test-slave-cbn3 krb5kdc[28969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980616, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:36:56.722728 29339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:56.705245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:46619 (local address 127.10.131.62:33639)
0620 18:36:56.705709 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:56.705713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:56.706731 (+ 1018us) server_negotiation.cc:407] Connection header received
0620 18:36:56.708511 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:56.708517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:56.708612 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:56.708727 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:56.710184 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:56.710978 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:56.712484 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:56.712974 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:56.713118 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:56.716685 (+ 3567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:56.716690 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:56.716693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:56.716724 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:56.719268 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:56.720153 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:56.720156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:56.720159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:56.720274 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:56.720728 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:56.720731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:56.720733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:56.721069 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:56.721240 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:56.722098 (+ 858us) server_negotiation.cc:299] Negotiation successful
0620 18:36:56.722348 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":244,"threads_started":1}
I20260620 18:36:56.723977 29340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:56.705870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:33639 (local address 127.10.131.2:46619)
0620 18:36:56.706580 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:56.706619 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:56.708258 (+ 1639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:56.708997 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:56.709005 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:56.709484 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:56.710016 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:56.710031 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:56.711127 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:56.711132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:56.712340 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:56.712352 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:56.713207 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:56.713216 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:56.713461 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:56.714132 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:56.714153 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:56.716441 (+ 2288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:56.719422 (+ 2981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:56.719433 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:56.719451 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:56.720044 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:56.720377 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:56.720384 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:56.720390 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:56.720627 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:56.721227 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:56.721237 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:56.721682 (+ 445us) client_negotiation.cc:769] Sending connection context
0620 18:36:56.721996 (+ 314us) client_negotiation.cc:240] Negotiation successful
0620 18:36:56.722583 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":320,"threads_started":1}
I20260620 18:36:56.725553 29337 heartbeater.cc:347] Connected to a master server at 127.10.131.62:33639
I20260620 18:36:56.725862 29337 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:56.726543 29337 heartbeater.cc:511] Master 127.10.131.62:33639 requested a full tablet report, sending...
I20260620 18:36:56.728061 29014 ts_manager.cc:194] Registered new tserver with Master: 95918c3a2807477ca6d9d16ec93b94ae (127.10.131.2:36943)
I20260620 18:36:56.729004 29014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.2@KRBTEST.COM'} at 127.10.131.2:46619
WARNING: no policy specified for kudu/127.10.131.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab.
Entry for principal kudu/127.10.131.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab.
I20260620 18:36:56.759737 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.3:0
--local_ip_for_outbound_sockets=127.10.131.3
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:33639
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:43747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:36:57.083382 29343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:57.083676 29343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:57.083860 29343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:57.090866 29343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:57.091013 29343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:57.091223 29343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.3
I20260620 18:36:57.101368 29343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:43747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:33639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.3
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:36:57.102586 29343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:57.103839 29343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:57.116577 29348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:57.116477 29349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:57.116511 29351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:57.122236 29343 server_base.cc:1034] running on GCE node
I20260620 18:36:57.122881 29343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:57.123759 29343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:57.124953 29343 hybrid_clock.cc:648] HybridClock initialized: now 1781980617124886 us; error 46 us; skew 500 ppm
I20260620 18:36:57.125280 29343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:36:57 dist-test-slave-cbn3 krb5kdc[28969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980617, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:36:57.128273 29343 init.cc:377] Logged in from keytab as kudu/127.10.131.3@KRBTEST.COM (short username kudu)
I20260620 18:36:57.129989 29343 webserver.cc:460] Webserver started at http://127.10.131.3:42157/ using document root <none> and password file <none>
I20260620 18:36:57.130722 29343 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:57.130863 29343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:57.131182 29343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:57.134073 29343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-2/data/instance:
uuid: "0001f8e3f3eb478fb932a0f9538ca7f1"
format_stamp: "Formatted at 2026-06-20 18:36:57 on dist-test-slave-cbn3"
server_key: "bdc3810ee524ced4643aa0ea237860d9"
server_key_iv: "8341743e4ddacc1b4d0ce264d20d4f47"
server_key_version: "encryptionkey@0"
I20260620 18:36:57.134836 29343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance:
uuid: "0001f8e3f3eb478fb932a0f9538ca7f1"
format_stamp: "Formatted at 2026-06-20 18:36:57 on dist-test-slave-cbn3"
server_key: "bdc3810ee524ced4643aa0ea237860d9"
server_key_iv: "8341743e4ddacc1b4d0ce264d20d4f47"
server_key_version: "encryptionkey@0"
I20260620 18:36:57.139961 29343 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:36:57.143316 29358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:57.144511 29343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:36:57.144692 29343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-2/wal
uuid: "0001f8e3f3eb478fb932a0f9538ca7f1"
format_stamp: "Formatted at 2026-06-20 18:36:57 on dist-test-slave-cbn3"
server_key: "bdc3810ee524ced4643aa0ea237860d9"
server_key_iv: "8341743e4ddacc1b4d0ce264d20d4f47"
server_key_version: "encryptionkey@0"
I20260620 18:36:57.144946 29343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:57.160444 29343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:57.164569 29343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:57.164878 29343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:57.165827 29343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:57.167163 29343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:57.167275 29343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:57.167450 29343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:57.167531 29343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:57.202382 29343 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.3:40939
I20260620 18:36:57.202420 29470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.3:40939 every 8 connection(s)
I20260620 18:36:57.203630 29343 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
I20260620 18:36:57.213897 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 29343
I20260620 18:36:57.214087 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance
I20260620 18:36:57.214500 29203 heartbeater.cc:503] Master 127.10.131.62:33639 was elected leader, sending a full tablet report...
I20260620 18:36:57.214687 10764 external_mini_cluster.cc:1428] Setting key 97e9ab24cf0ee4fe4e108ac009524af3
Jun 20 18:36:57 dist-test-slave-cbn3 krb5kdc[28969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980617, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:36:57.222922 29339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:57.206323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:54717 (local address 127.10.131.62:33639)
0620 18:36:57.206502 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:57.206506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:57.207424 (+ 918us) server_negotiation.cc:407] Connection header received
0620 18:36:57.209150 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:57.209154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:57.209235 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:57.209323 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:57.210632 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.211363 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.212949 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.213363 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.213455 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:57.217412 (+ 3957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:57.217417 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:57.217420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:57.217472 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:57.219857 (+ 2385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:57.220576 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:57.220581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:57.220584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:57.220709 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:57.221184 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:57.221189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:57.221193 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:57.221542 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:57.221703 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:57.222364 (+ 661us) server_negotiation.cc:299] Negotiation successful
0620 18:36:57.222585 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 18:36:57.223839 29473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:57.206620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:33639 (local address 127.10.131.3:54717)
0620 18:36:57.207252 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:57.207292 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:57.208897 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:57.209508 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:57.209515 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:57.209981 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:57.210490 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:57.210504 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.211493 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.211496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:57.212686 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:57.212696 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.213545 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.213550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:57.213729 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:57.214480 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:36:57.214510 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:57.217204 (+ 2694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:57.220001 (+ 2797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:57.220010 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:57.220026 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:57.220451 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:57.220818 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:36:57.220823 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:36:57.220826 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:36:57.221069 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:36:57.221668 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:57.221675 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:57.222010 (+ 335us) client_negotiation.cc:769] Sending connection context
0620 18:36:57.222271 (+ 261us) client_negotiation.cc:240] Negotiation successful
0620 18:36:57.222805 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":250,"threads_started":1}
I20260620 18:36:57.225274 29471 heartbeater.cc:347] Connected to a master server at 127.10.131.62:33639
I20260620 18:36:57.225562 29471 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:57.226254 29471 heartbeater.cc:511] Master 127.10.131.62:33639 requested a full tablet report, sending...
I20260620 18:36:57.227563 29014 ts_manager.cc:194] Registered new tserver with Master: 0001f8e3f3eb478fb932a0f9538ca7f1 (127.10.131.3:40939)
I20260620 18:36:57.228652 29014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.3@KRBTEST.COM'} at 127.10.131.3:54717
I20260620 18:36:57.230001 10764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:36:57.247484 29339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:57.233928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54250 (local address 127.10.131.62:33639)
0620 18:36:57.234133 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:57.234141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:57.234536 (+ 395us) server_negotiation.cc:407] Connection header received
0620 18:36:57.234729 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:57.234733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:57.234798 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:57.234899 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:57.236281 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.237064 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.238523 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.238955 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.239084 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:57.241935 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:57.241942 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:57.241947 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:57.241991 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:57.244397 (+ 2406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:57.245278 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:57.245283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:57.245287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:57.245410 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:57.245895 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:57.245900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:57.245905 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:57.246317 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:57.246623 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:57.246994 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 18:36:57.247173 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:36:57.251369 29014 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:36:57.258090 29014 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:36:57.281328 29484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:57.270272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48072 (local address 127.10.131.1:43403)
0620 18:36:57.273603 (+ 3331us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:57.273612 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:57.273639 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:36:57.273735 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:57.273743 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:57.274012 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:57.274846 (+ 834us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:57.275115 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.276648 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.279247 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.280136 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.280289 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:57.280521 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:57.280800 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 18:36:57.280979 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3163,"thread_start_us":2857,"threads_started":1}
I20260620 18:36:57.281400 29483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:57.269950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:43403 (local address 127.0.0.1:48072)
0620 18:36:57.271809 (+ 1859us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:57.271867 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:57.272069 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:57.274256 (+ 2187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:57.274264 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:57.274303 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:57.274786 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:57.274801 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.277004 (+ 2203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.277009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:57.279086 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:57.279097 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.280367 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.280373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:57.280554 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:57.280599 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 18:36:57.280751 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 18:36:57.280955 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1540,"mutex_wait_us":3,"thread_start_us":1386,"threads_started":1}
I20260620 18:36:57.283985 29487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:57.272811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:40939 (local address 127.0.0.1:46942)
0620 18:36:57.275326 (+ 2515us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:57.275362 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:57.275472 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:57.276101 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:57.276106 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:57.276133 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:57.276603 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:57.276615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.278342 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.278349 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:57.280781 (+ 2432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:57.280789 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.283125 (+ 2336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.283131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:57.283299 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:57.283329 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 18:36:57.283422 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 18:36:57.283540 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2407,"mutex_wait_us":66,"thread_start_us":2246,"threads_started":1}
I20260620 18:36:57.284052 29488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:57.273545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46942 (local address 127.10.131.3:40939)
0620 18:36:57.274734 (+ 1189us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:57.274743 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:57.275587 (+ 844us) server_negotiation.cc:407] Connection header received
0620 18:36:57.275684 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:57.275691 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:57.275913 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:57.276078 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:57.276792 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.278202 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.281587 (+ 3385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.282640 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.282712 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:57.282943 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:57.283494 (+ 551us) server_negotiation.cc:299] Negotiation successful
0620 18:36:57.283688 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1002,"thread_start_us":829,"threads_started":1}
I20260620 18:36:57.284750 29485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:57.270383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:36943 (local address 127.0.0.1:39906)
0620 18:36:57.272705 (+ 2322us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:57.272725 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:57.272875 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:57.276154 (+ 3279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:57.276159 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:57.276186 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:57.276650 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:57.276664 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.278544 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.278551 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:57.280767 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:57.280783 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.283714 (+ 2931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.283721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:57.283906 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:57.283941 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 18:36:57.284007 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 18:36:57.284373 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2173,"mutex_wait_us":14,"thread_start_us":2029,"threads_started":1}
I20260620 18:36:57.285398 29486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:57.270809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39906 (local address 127.10.131.2:36943)
0620 18:36:57.275681 (+ 4872us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:57.275687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:57.275710 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:36:57.275783 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:57.275788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:57.275955 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:57.276507 (+ 552us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:57.276864 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.278401 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.280978 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.283463 (+ 2485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.284653 (+ 1190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:57.284842 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:57.284939 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 18:36:57.285083 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4729,"thread_start_us":4619,"threads_started":1}
I20260620 18:36:57.285863 29138 tablet_service.cc:1467] Processing CreateTablet for tablet b37b2922f7f54956b59cfa5fa2286187 (DEFAULT_TABLE table=test-table [id=e26398b6fc1b45879902029263c705cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:57.287634 29406 tablet_service.cc:1467] Processing CreateTablet for tablet b37b2922f7f54956b59cfa5fa2286187 (DEFAULT_TABLE table=test-table [id=e26398b6fc1b45879902029263c705cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:57.288215 29138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b37b2922f7f54956b59cfa5fa2286187. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:57.288818 29406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b37b2922f7f54956b59cfa5fa2286187. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:57.288523 29272 tablet_service.cc:1467] Processing CreateTablet for tablet b37b2922f7f54956b59cfa5fa2286187 (DEFAULT_TABLE table=test-table [id=e26398b6fc1b45879902029263c705cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:36:57.289861 29272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b37b2922f7f54956b59cfa5fa2286187. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:57.300981 29489 tablet_bootstrap.cc:492] T b37b2922f7f54956b59cfa5fa2286187 P 0001f8e3f3eb478fb932a0f9538ca7f1: Bootstrap starting.
I20260620 18:36:57.302695 29490 tablet_bootstrap.cc:492] T b37b2922f7f54956b59cfa5fa2286187 P f889374377284726bca487bd8bdc6ccd: Bootstrap starting.
I20260620 18:36:57.303548 29491 tablet_bootstrap.cc:492] T b37b2922f7f54956b59cfa5fa2286187 P 95918c3a2807477ca6d9d16ec93b94ae: Bootstrap starting.
I20260620 18:36:57.304358 29489 tablet_bootstrap.cc:654] T b37b2922f7f54956b59cfa5fa2286187 P 0001f8e3f3eb478fb932a0f9538ca7f1: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:57.305666 29489 log.cc:824] T b37b2922f7f54956b59cfa5fa2286187 P 0001f8e3f3eb478fb932a0f9538ca7f1: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:57.306417 29491 tablet_bootstrap.cc:654] T b37b2922f7f54956b59cfa5fa2286187 P 95918c3a2807477ca6d9d16ec93b94ae: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:57.306694 29490 tablet_bootstrap.cc:654] T b37b2922f7f54956b59cfa5fa2286187 P f889374377284726bca487bd8bdc6ccd: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:57.307476 29491 log.cc:824] T b37b2922f7f54956b59cfa5fa2286187 P 95918c3a2807477ca6d9d16ec93b94ae: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:57.307878 29490 log.cc:824] T b37b2922f7f54956b59cfa5fa2286187 P f889374377284726bca487bd8bdc6ccd: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:57.309696 29489 tablet_bootstrap.cc:492] T b37b2922f7f54956b59cfa5fa2286187 P 0001f8e3f3eb478fb932a0f9538ca7f1: No bootstrap required, opened a new log
I20260620 18:36:57.309976 29489 ts_tablet_manager.cc:1397] T b37b2922f7f54956b59cfa5fa2286187 P 0001f8e3f3eb478fb932a0f9538ca7f1: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260620 18:36:57.310410 29491 tablet_bootstrap.cc:492] T b37b2922f7f54956b59cfa5fa2286187 P 95918c3a2807477ca6d9d16ec93b94ae: No bootstrap required, opened a new log
I20260620 18:36:57.310669 29491 ts_tablet_manager.cc:1397] T b37b2922f7f54956b59cfa5fa2286187 P 95918c3a2807477ca6d9d16ec93b94ae: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 18:36:57.311100 29490 tablet_bootstrap.cc:492] T b37b2922f7f54956b59cfa5fa2286187 P f889374377284726bca487bd8bdc6ccd: No bootstrap required, opened a new log
I20260620 18:36:57.311348 29490 ts_tablet_manager.cc:1397] T b37b2922f7f54956b59cfa5fa2286187 P f889374377284726bca487bd8bdc6ccd: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260620 18:36:57.317744 29491 raft_consensus.cc:348] T b37b2922f7f54956b59cfa5fa2286187 P 95918c3a2807477ca6d9d16ec93b94ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f889374377284726bca487bd8bdc6ccd" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 43403 } } peers { permanent_uuid: "95918c3a2807477ca6d9d16ec93b94ae" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36943 } } peers { permanent_uuid: "0001f8e3f3eb478fb932a0f9538ca7f1" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40939 } }
I20260620 18:36:57.318140 29491 raft_consensus.cc:374] T b37b2922f7f54956b59cfa5fa2286187 P 95918c3a2807477ca6d9d16ec93b94ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:57.318229 29491 raft_consensus.cc:729] T b37b2922f7f54956b59cfa5fa2286187 P 95918c3a2807477ca6d9d16ec93b94ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95918c3a2807477ca6d9d16ec93b94ae, State: Initialized, Role: FOLLOWER
I20260620 18:36:57.318842 29491 consensus_queue.cc:260] T b37b2922f7f54956b59cfa5fa2286187 P 95918c3a2807477ca6d9d16ec93b94ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f889374377284726bca487bd8bdc6ccd" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 43403 } } peers { permanent_uuid: "95918c3a2807477ca6d9d16ec93b94ae" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36943 } } peers { permanent_uuid: "0001f8e3f3eb478fb932a0f9538ca7f1" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40939 } }
I20260620 18:36:57.320693 29489 raft_consensus.cc:348] T b37b2922f7f54956b59cfa5fa2286187 P 0001f8e3f3eb478fb932a0f9538ca7f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f889374377284726bca487bd8bdc6ccd" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 43403 } } peers { permanent_uuid: "95918c3a2807477ca6d9d16ec93b94ae" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36943 } } peers { permanent_uuid: "0001f8e3f3eb478fb932a0f9538ca7f1" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40939 } }
I20260620 18:36:57.321192 29489 raft_consensus.cc:374] T b37b2922f7f54956b59cfa5fa2286187 P 0001f8e3f3eb478fb932a0f9538ca7f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:57.321307 29489 raft_consensus.cc:729] T b37b2922f7f54956b59cfa5fa2286187 P 0001f8e3f3eb478fb932a0f9538ca7f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0001f8e3f3eb478fb932a0f9538ca7f1, State: Initialized, Role: FOLLOWER
I20260620 18:36:57.321810 29337 heartbeater.cc:503] Master 127.10.131.62:33639 was elected leader, sending a full tablet report...
I20260620 18:36:57.321712 29490 raft_consensus.cc:348] T b37b2922f7f54956b59cfa5fa2286187 P f889374377284726bca487bd8bdc6ccd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f889374377284726bca487bd8bdc6ccd" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 43403 } } peers { permanent_uuid: "95918c3a2807477ca6d9d16ec93b94ae" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36943 } } peers { permanent_uuid: "0001f8e3f3eb478fb932a0f9538ca7f1" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40939 } }
I20260620 18:36:57.322124 29490 raft_consensus.cc:374] T b37b2922f7f54956b59cfa5fa2286187 P f889374377284726bca487bd8bdc6ccd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:57.322216 29490 raft_consensus.cc:729] T b37b2922f7f54956b59cfa5fa2286187 P f889374377284726bca487bd8bdc6ccd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f889374377284726bca487bd8bdc6ccd, State: Initialized, Role: FOLLOWER
I20260620 18:36:57.322058 29489 consensus_queue.cc:260] T b37b2922f7f54956b59cfa5fa2286187 P 0001f8e3f3eb478fb932a0f9538ca7f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f889374377284726bca487bd8bdc6ccd" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 43403 } } peers { permanent_uuid: "95918c3a2807477ca6d9d16ec93b94ae" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36943 } } peers { permanent_uuid: "0001f8e3f3eb478fb932a0f9538ca7f1" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40939 } }
I20260620 18:36:57.322872 29490 consensus_queue.cc:260] T b37b2922f7f54956b59cfa5fa2286187 P f889374377284726bca487bd8bdc6ccd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f889374377284726bca487bd8bdc6ccd" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 43403 } } peers { permanent_uuid: "95918c3a2807477ca6d9d16ec93b94ae" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36943 } } peers { permanent_uuid: "0001f8e3f3eb478fb932a0f9538ca7f1" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40939 } }
I20260620 18:36:57.324712 29490 ts_tablet_manager.cc:1428] T b37b2922f7f54956b59cfa5fa2286187 P f889374377284726bca487bd8bdc6ccd: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260620 18:36:57.325351 29491 ts_tablet_manager.cc:1428] T b37b2922f7f54956b59cfa5fa2286187 P 95918c3a2807477ca6d9d16ec93b94ae: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260620 18:36:57.329566 29471 heartbeater.cc:503] Master 127.10.131.62:33639 was elected leader, sending a full tablet report...
I20260620 18:36:57.330226 29489 ts_tablet_manager.cc:1428] T b37b2922f7f54956b59cfa5fa2286187 P 0001f8e3f3eb478fb932a0f9538ca7f1: Time spent starting tablet: real 0.020s user 0.009s sys 0.009s
W20260620 18:36:57.432549 29204 tablet.cc:2369] T b37b2922f7f54956b59cfa5fa2286187 P f889374377284726bca487bd8bdc6ccd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:36:57.454957 29338 tablet.cc:2369] T b37b2922f7f54956b59cfa5fa2286187 P 95918c3a2807477ca6d9d16ec93b94ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:36:57.455562 29472 tablet.cc:2369] T b37b2922f7f54956b59cfa5fa2286187 P 0001f8e3f3eb478fb932a0f9538ca7f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:36:57.699848 29497 raft_consensus.cc:482] T b37b2922f7f54956b59cfa5fa2286187 P f889374377284726bca487bd8bdc6ccd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:36:57.700141 29497 raft_consensus.cc:504] T b37b2922f7f54956b59cfa5fa2286187 P f889374377284726bca487bd8bdc6ccd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f889374377284726bca487bd8bdc6ccd" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 43403 } } peers { permanent_uuid: "95918c3a2807477ca6d9d16ec93b94ae" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36943 } } peers { permanent_uuid: "0001f8e3f3eb478fb932a0f9538ca7f1" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40939 } }
I20260620 18:36:57.702226 29497 leader_election.cc:288] T b37b2922f7f54956b59cfa5fa2286187 P f889374377284726bca487bd8bdc6ccd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95918c3a2807477ca6d9d16ec93b94ae (127.10.131.2:36943), 0001f8e3f3eb478fb932a0f9538ca7f1 (127.10.131.3:40939)
I20260620 18:36:57.709612 29488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:57.703791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:43811 (local address 127.10.131.3:40939)
0620 18:36:57.703952 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:57.703958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:57.704537 (+ 579us) server_negotiation.cc:407] Connection header received
0620 18:36:57.704730 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:57.704738 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:57.704818 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:57.704925 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:57.705413 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.706109 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.707803 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.708631 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.708801 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:57.708900 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:57.709202 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 18:36:57.709348 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:36:57.709645 29499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:57.703651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:40939 (local address 127.10.131.1:43811)
0620 18:36:57.704386 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:57.704403 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:57.704567 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:57.704961 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:57.704965 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:57.704987 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:57.705273 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:57.705281 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.706271 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.706276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:57.707653 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:57.707664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.708833 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.708860 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:57.709017 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:57.709062 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 18:36:57.709158 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 18:36:57.709306 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":448,"threads_started":1}
I20260620 18:36:57.710920 29486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:57.702873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:55967 (local address 127.10.131.2:36943)
0620 18:36:57.703108 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:57.703116 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:57.703616 (+ 500us) server_negotiation.cc:407] Connection header received
0620 18:36:57.703932 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:57.703942 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:57.704041 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:57.704117 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:36:57.704942 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.706020 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.708619 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.709971 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.710149 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:57.710215 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:57.710545 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 18:36:57.710669 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:36:57.710848 29426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b37b2922f7f54956b59cfa5fa2286187" candidate_uuid: "f889374377284726bca487bd8bdc6ccd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0001f8e3f3eb478fb932a0f9538ca7f1" is_pre_election: true
I20260620 18:36:57.711127 29498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:57.702755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:36943 (local address 127.10.131.1:55967)
0620 18:36:57.703516 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:57.703537 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:57.703735 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:57.704337 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:57.704341 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:57.704362 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:36:57.704762 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:57.704773 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.706196 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.706201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:57.708445 (+ 2244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:57.708459 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.710173 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.710179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:57.710360 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:57.710401 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 18:36:57.710496 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 18:36:57.710626 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"mutex_wait_us":1,"thread_start_us":425,"threads_started":1}
I20260620 18:36:57.711387 29426 raft_consensus.cc:2454] T b37b2922f7f54956b59cfa5fa2286187 P 0001f8e3f3eb478fb932a0f9538ca7f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f889374377284726bca487bd8bdc6ccd in term 0.
I20260620 18:36:57.712103 29092 leader_election.cc:302] T b37b2922f7f54956b59cfa5fa2286187 P f889374377284726bca487bd8bdc6ccd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0001f8e3f3eb478fb932a0f9538ca7f1, f889374377284726bca487bd8bdc6ccd; no voters:
I20260620 18:36:57.712500 29497 raft_consensus.cc:2781] T b37b2922f7f54956b59cfa5fa2286187 P f889374377284726bca487bd8bdc6ccd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:36:57.712343 29292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b37b2922f7f54956b59cfa5fa2286187" candidate_uuid: "f889374377284726bca487bd8bdc6ccd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95918c3a2807477ca6d9d16ec93b94ae" is_pre_election: true
I20260620 18:36:57.712653 29497 raft_consensus.cc:482] T b37b2922f7f54956b59cfa5fa2286187 P f889374377284726bca487bd8bdc6ccd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:36:57.712762 29497 raft_consensus.cc:3037] T b37b2922f7f54956b59cfa5fa2286187 P f889374377284726bca487bd8bdc6ccd [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:57.712905 29292 raft_consensus.cc:2454] T b37b2922f7f54956b59cfa5fa2286187 P 95918c3a2807477ca6d9d16ec93b94ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f889374377284726bca487bd8bdc6ccd in term 0.
I20260620 18:36:57.715024 29497 raft_consensus.cc:504] T b37b2922f7f54956b59cfa5fa2286187 P f889374377284726bca487bd8bdc6ccd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f889374377284726bca487bd8bdc6ccd" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 43403 } } peers { permanent_uuid: "95918c3a2807477ca6d9d16ec93b94ae" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36943 } } peers { permanent_uuid: "0001f8e3f3eb478fb932a0f9538ca7f1" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40939 } }
I20260620 18:36:57.715595 29497 leader_election.cc:288] T b37b2922f7f54956b59cfa5fa2286187 P f889374377284726bca487bd8bdc6ccd [CANDIDATE]: Term 1 election: Requested vote from peers 95918c3a2807477ca6d9d16ec93b94ae (127.10.131.2:36943), 0001f8e3f3eb478fb932a0f9538ca7f1 (127.10.131.3:40939)
I20260620 18:36:57.716131 29292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b37b2922f7f54956b59cfa5fa2286187" candidate_uuid: "f889374377284726bca487bd8bdc6ccd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95918c3a2807477ca6d9d16ec93b94ae"
I20260620 18:36:57.716144 29426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b37b2922f7f54956b59cfa5fa2286187" candidate_uuid: "f889374377284726bca487bd8bdc6ccd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0001f8e3f3eb478fb932a0f9538ca7f1"
I20260620 18:36:57.716321 29292 raft_consensus.cc:3037] T b37b2922f7f54956b59cfa5fa2286187 P 95918c3a2807477ca6d9d16ec93b94ae [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:57.716331 29426 raft_consensus.cc:3037] T b37b2922f7f54956b59cfa5fa2286187 P 0001f8e3f3eb478fb932a0f9538ca7f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:57.718611 29292 raft_consensus.cc:2454] T b37b2922f7f54956b59cfa5fa2286187 P 95918c3a2807477ca6d9d16ec93b94ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f889374377284726bca487bd8bdc6ccd in term 1.
I20260620 18:36:57.718783 29426 raft_consensus.cc:2454] T b37b2922f7f54956b59cfa5fa2286187 P 0001f8e3f3eb478fb932a0f9538ca7f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f889374377284726bca487bd8bdc6ccd in term 1.
I20260620 18:36:57.719060 29092 leader_election.cc:302] T b37b2922f7f54956b59cfa5fa2286187 P f889374377284726bca487bd8bdc6ccd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95918c3a2807477ca6d9d16ec93b94ae, f889374377284726bca487bd8bdc6ccd; no voters:
I20260620 18:36:57.719367 29497 raft_consensus.cc:2781] T b37b2922f7f54956b59cfa5fa2286187 P f889374377284726bca487bd8bdc6ccd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:36:57.719801 29497 raft_consensus.cc:686] T b37b2922f7f54956b59cfa5fa2286187 P f889374377284726bca487bd8bdc6ccd [term 1 LEADER]: Becoming Leader. State: Replica: f889374377284726bca487bd8bdc6ccd, State: Running, Role: LEADER
I20260620 18:36:57.720250 29497 consensus_queue.cc:237] T b37b2922f7f54956b59cfa5fa2286187 P f889374377284726bca487bd8bdc6ccd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f889374377284726bca487bd8bdc6ccd" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 43403 } } peers { permanent_uuid: "95918c3a2807477ca6d9d16ec93b94ae" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36943 } } peers { permanent_uuid: "0001f8e3f3eb478fb932a0f9538ca7f1" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40939 } }
I20260620 18:36:57.724888 29014 catalog_manager.cc:5495] T b37b2922f7f54956b59cfa5fa2286187 P f889374377284726bca487bd8bdc6ccd reported cstate change: term changed from 0 to 1, leader changed from <none> to f889374377284726bca487bd8bdc6ccd (127.10.131.1). New cstate: current_term: 1 leader_uuid: "f889374377284726bca487bd8bdc6ccd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f889374377284726bca487bd8bdc6ccd" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 43403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95918c3a2807477ca6d9d16ec93b94ae" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0001f8e3f3eb478fb932a0f9538ca7f1" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40939 } health_report { overall_health: UNKNOWN } } }
I20260620 18:36:57.849356 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:57.841557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48086 (local address 127.10.131.1:43403)
0620 18:36:57.842335 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:57.842340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:57.842357 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:36:57.842532 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:57.842536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:57.842628 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:57.842768 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:36:57.843388 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.844057 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.846654 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.847092 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.847207 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:57.847932 (+ 725us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:36:57.848475 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:36:57.848650 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:57.848893 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 18:36:57.848954 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":572,"threads_started":1}
I20260620 18:36:57.858143 29292 raft_consensus.cc:1261] T b37b2922f7f54956b59cfa5fa2286187 P 95918c3a2807477ca6d9d16ec93b94ae [term 1 FOLLOWER]: Refusing update from remote peer f889374377284726bca487bd8bdc6ccd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:36:57.858336 29426 raft_consensus.cc:1261] T b37b2922f7f54956b59cfa5fa2286187 P 0001f8e3f3eb478fb932a0f9538ca7f1 [term 1 FOLLOWER]: Refusing update from remote peer f889374377284726bca487bd8bdc6ccd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:36:57.859154 29497 consensus_queue.cc:1035] T b37b2922f7f54956b59cfa5fa2286187 P f889374377284726bca487bd8bdc6ccd [LEADER]: Connected to new peer: Peer: permanent_uuid: "95918c3a2807477ca6d9d16ec93b94ae" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 36943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:36:57.859476 29500 consensus_queue.cc:1035] T b37b2922f7f54956b59cfa5fa2286187 P f889374377284726bca487bd8bdc6ccd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0001f8e3f3eb478fb932a0f9538ca7f1" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 40939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:36:57.879650 29505 mvcc.cc:205] Tried to move back new op lower bound from 7298992610735816704 to 7298992610186223616. Current Snapshot: MvccSnapshot[applied={T|T < 7298992610735816704}]
I20260620 18:36:57.882218 29509 mvcc.cc:205] Tried to move back new op lower bound from 7298992610735816704 to 7298992610186223616. Current Snapshot: MvccSnapshot[applied={T|T < 7298992610735816704}]
I20260620 18:36:57.907119 29014 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:36:57.907430 29014 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:36:57.911938 29014 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e4745180ec054466936fe98e736c8532: Sending DeleteTablet for 3 replicas of tablet b37b2922f7f54956b59cfa5fa2286187
I20260620 18:36:57.913357 29272 tablet_service.cc:1514] Processing DeleteTablet for tablet b37b2922f7f54956b59cfa5fa2286187 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:57 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:39906
I20260620 18:36:57.913331 29138 tablet_service.cc:1514] Processing DeleteTablet for tablet b37b2922f7f54956b59cfa5fa2286187 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:57 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:48072
I20260620 18:36:57.913868 29406 tablet_service.cc:1514] Processing DeleteTablet for tablet b37b2922f7f54956b59cfa5fa2286187 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:36:57 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:46942
I20260620 18:36:57.915791 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 29075
I20260620 18:36:57.945936 29518 tablet_replica.cc:330] T b37b2922f7f54956b59cfa5fa2286187 P f889374377284726bca487bd8bdc6ccd: stopping tablet replica
I20260620 18:36:57.946741 29518 raft_consensus.cc:2229] T b37b2922f7f54956b59cfa5fa2286187 P f889374377284726bca487bd8bdc6ccd [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:36:57.952391 29518 raft_consensus.cc:2258] T b37b2922f7f54956b59cfa5fa2286187 P f889374377284726bca487bd8bdc6ccd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:57.953356 29519 tablet_replica.cc:330] T b37b2922f7f54956b59cfa5fa2286187 P 0001f8e3f3eb478fb932a0f9538ca7f1: stopping tablet replica
I20260620 18:36:57.953965 29519 raft_consensus.cc:2229] T b37b2922f7f54956b59cfa5fa2286187 P 0001f8e3f3eb478fb932a0f9538ca7f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:36:57.954550 29519 raft_consensus.cc:2258] T b37b2922f7f54956b59cfa5fa2286187 P 0001f8e3f3eb478fb932a0f9538ca7f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:36:57.955164 29518 ts_tablet_manager.cc:1905] T b37b2922f7f54956b59cfa5fa2286187 P f889374377284726bca487bd8bdc6ccd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:57.955245 29517 tablet_replica.cc:330] T b37b2922f7f54956b59cfa5fa2286187 P 95918c3a2807477ca6d9d16ec93b94ae: stopping tablet replica
I20260620 18:36:57.955674 29517 raft_consensus.cc:2229] T b37b2922f7f54956b59cfa5fa2286187 P 95918c3a2807477ca6d9d16ec93b94ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:36:57.956748 29519 ts_tablet_manager.cc:1905] T b37b2922f7f54956b59cfa5fa2286187 P 0001f8e3f3eb478fb932a0f9538ca7f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:57.959105 29517 raft_consensus.cc:2258] T b37b2922f7f54956b59cfa5fa2286187 P 95918c3a2807477ca6d9d16ec93b94ae [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 20 18:36:57 dist-test-slave-cbn3 krb5kdc[28969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.1@KRBTEST.COM
I20260620 18:36:57.963852 29518 ts_tablet_manager.cc:1918] T b37b2922f7f54956b59cfa5fa2286187 P f889374377284726bca487bd8bdc6ccd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:36:57.964052 29518 log.cc:1196] T b37b2922f7f54956b59cfa5fa2286187 P f889374377284726bca487bd8bdc6ccd: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-0/wal/wals/b37b2922f7f54956b59cfa5fa2286187
I20260620 18:36:57.964632 29518 ts_tablet_manager.cc:1939] T b37b2922f7f54956b59cfa5fa2286187 P f889374377284726bca487bd8bdc6ccd: Deleting consensus metadata
I20260620 18:36:57.965829 29517 ts_tablet_manager.cc:1905] T b37b2922f7f54956b59cfa5fa2286187 P 95918c3a2807477ca6d9d16ec93b94ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:36:57.966576 29001 catalog_manager.cc:4841] TS f889374377284726bca487bd8bdc6ccd (127.10.131.1:43403): tablet b37b2922f7f54956b59cfa5fa2286187 (table test-table [id=e26398b6fc1b45879902029263c705cd]) successfully deleted
I20260620 18:36:57.970043 29519 ts_tablet_manager.cc:1918] T b37b2922f7f54956b59cfa5fa2286187 P 0001f8e3f3eb478fb932a0f9538ca7f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:36:57.970320 29519 log.cc:1196] T b37b2922f7f54956b59cfa5fa2286187 P 0001f8e3f3eb478fb932a0f9538ca7f1: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-2/wal/wals/b37b2922f7f54956b59cfa5fa2286187
I20260620 18:36:57.970938 29519 ts_tablet_manager.cc:1939] T b37b2922f7f54956b59cfa5fa2286187 P 0001f8e3f3eb478fb932a0f9538ca7f1: Deleting consensus metadata
I20260620 18:36:57.976527 28999 catalog_manager.cc:4841] TS 0001f8e3f3eb478fb932a0f9538ca7f1 (127.10.131.3:40939): tablet b37b2922f7f54956b59cfa5fa2286187 (table test-table [id=e26398b6fc1b45879902029263c705cd]) successfully deleted
I20260620 18:36:57.976898 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:57.917365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48098 (local address 127.10.131.1:43403)
0620 18:36:57.917525 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:57.917530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:57.942084 (+ 24554us) server_negotiation.cc:407] Connection header received
0620 18:36:57.942667 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:57.942677 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:57.942807 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:57.942918 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:57.947528 (+ 4610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.949078 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.953032 (+ 3954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:57.953734 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:57.953846 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:57.968198 (+ 14352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:57.968213 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:57.968232 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:57.968289 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:57.972410 (+ 4121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:57.973446 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:57.973456 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:57.973459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:57.973639 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:57.974141 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:57.974146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:57.974150 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:57.974598 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:57.974907 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:57.976139 (+ 1232us) server_negotiation.cc:299] Negotiation successful
0620 18:36:57.976385 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 18:36:57.977965 29198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:57.980535 29517 ts_tablet_manager.cc:1918] T b37b2922f7f54956b59cfa5fa2286187 P 95918c3a2807477ca6d9d16ec93b94ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:36:57.980851 29517 log.cc:1196] T b37b2922f7f54956b59cfa5fa2286187 P 95918c3a2807477ca6d9d16ec93b94ae: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781980314095189-10764-0/minicluster-data/ts-1/wal/wals/b37b2922f7f54956b59cfa5fa2286187
I20260620 18:36:57.981537 29517 ts_tablet_manager.cc:1939] T b37b2922f7f54956b59cfa5fa2286187 P 95918c3a2807477ca6d9d16ec93b94ae: Deleting consensus metadata
I20260620 18:36:57.983731 28999 catalog_manager.cc:4841] TS 95918c3a2807477ca6d9d16ec93b94ae (127.10.131.2:36943): tablet b37b2922f7f54956b59cfa5fa2286187 (table test-table [id=e26398b6fc1b45879902029263c705cd]) successfully deleted
I20260620 18:36:58.089975 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 29075
I20260620 18:36:58.108246 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 29209
Jun 20 18:36:58 dist-test-slave-cbn3 krb5kdc[28969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.2@KRBTEST.COM
I20260620 18:36:58.123545 29486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:58.109126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39922 (local address 127.10.131.2:36943)
0620 18:36:58.109331 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:58.109337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:58.109358 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:36:58.109534 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:58.109539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:58.109599 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:58.109713 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:58.110911 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:58.111676 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:58.113067 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:58.113481 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:58.113614 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:58.117834 (+ 4220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:58.117848 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:58.117862 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:58.117894 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:58.120721 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:58.121518 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:58.121525 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:58.121527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:58.121639 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:58.122019 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:58.122027 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:58.122029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:58.122435 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:58.122712 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:58.123012 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 18:36:58.123233 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:36:58.124184 29332 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:58.213019 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 29209
I20260620 18:36:58.230396 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 29343
Jun 20 18:36:58 dist-test-slave-cbn3 krb5kdc[28969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.3@KRBTEST.COM
I20260620 18:36:58.246075 29523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:58.231143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46958 (local address 127.10.131.3:40939)
0620 18:36:58.231612 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:58.231617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:58.231631 (+ 14us) server_negotiation.cc:407] Connection header received
0620 18:36:58.231683 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:58.231687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:58.231744 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:58.231864 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:58.233238 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:58.233961 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:58.235344 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:58.235741 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:58.235866 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:58.240474 (+ 4608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:58.240488 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:58.240504 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:36:58.240542 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:58.243053 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:58.243693 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:58.243708 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:58.243710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:58.243850 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:36:58.244216 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:36:58.244223 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:36:58.244230 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:36:58.244582 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:58.244928 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:58.245499 (+ 571us) server_negotiation.cc:299] Negotiation successful
0620 18:36:58.245718 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":232,"threads_started":1}
I20260620 18:36:58.246760 29466 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:58.335000 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 29343
I20260620 18:36:58.351490 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 28982
I20260620 18:36:58.352523 29044 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:36:58.462393 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 28982
2026-06-20T18:36:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4656 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T18:36:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:36:58Z Disabled control of system clock
I20260620 18:36:58.536939 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:37471
--webserver_interface=127.10.131.62
--webserver_port=0
--builtin_ntp_servers=127.10.131.20:46649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:37471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:36:58.870929 29529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:58.871263 29529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:58.871449 29529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:58.879465 29529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:36:58.879609 29529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:58.879688 29529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:58.879788 29529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:36:58.879884 29529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:36:58.890206 29529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:46649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:37471
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:37471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:36:58.891568 29529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:58.893536 29529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:58.906587 29535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:58.906677 29537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:58.906941 29529 server_base.cc:1034] running on GCE node
W20260620 18:36:58.906842 29534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:58.907963 29529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:58.909464 29529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:58.910666 29529 hybrid_clock.cc:648] HybridClock initialized: now 1781980618910628 us; error 28 us; skew 500 ppm
I20260620 18:36:58.910976 29529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:36:58.917621 29529 webserver.cc:460] Webserver started at http://127.10.131.62:38189/ using document root <none> and password file <none>
I20260620 18:36:58.918354 29529 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:58.918444 29529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:58.918727 29529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:58.921600 29529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "fb2abb31e70d45568a66963cb091c970"
format_stamp: "Formatted at 2026-06-20 18:36:58 on dist-test-slave-cbn3"
server_key: "16303ddf5f9d611a79816beacdf96aa2"
server_key_iv: "085675570ac571c9f35c2f8466b07549"
server_key_version: "encryptionkey@0"
I20260620 18:36:58.922338 29529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "fb2abb31e70d45568a66963cb091c970"
format_stamp: "Formatted at 2026-06-20 18:36:58 on dist-test-slave-cbn3"
server_key: "16303ddf5f9d611a79816beacdf96aa2"
server_key_iv: "085675570ac571c9f35c2f8466b07549"
server_key_version: "encryptionkey@0"
I20260620 18:36:58.927233 29529 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:36:58.930487 29543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:58.931741 29529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:36:58.931878 29529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "fb2abb31e70d45568a66963cb091c970"
format_stamp: "Formatted at 2026-06-20 18:36:58 on dist-test-slave-cbn3"
server_key: "16303ddf5f9d611a79816beacdf96aa2"
server_key_iv: "085675570ac571c9f35c2f8466b07549"
server_key_version: "encryptionkey@0"
I20260620 18:36:58.932045 29529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:58.959754 29529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:58.960582 29529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:58.960817 29529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:58.979060 29529 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:37471
I20260620 18:36:58.979080 29594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:37471 every 8 connection(s)
I20260620 18:36:58.980451 29529 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:36:58.984123 29595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:36:58.989656 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 29529
I20260620 18:36:58.989830 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:36:58.990305 10764 external_mini_cluster.cc:1428] Setting key 3c1a17f575b74b3053ab41c0e7d34088
I20260620 18:36:58.995134 29595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb2abb31e70d45568a66963cb091c970: Bootstrap starting.
I20260620 18:36:58.998589 29595 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb2abb31e70d45568a66963cb091c970: Neither blocks nor log segments found. Creating new log.
I20260620 18:36:58.999837 29595 log.cc:824] T 00000000000000000000000000000000 P fb2abb31e70d45568a66963cb091c970: Log is configured to *not* fsync() on all Append() calls
I20260620 18:36:59.003080 29595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb2abb31e70d45568a66963cb091c970: No bootstrap required, opened a new log
I20260620 18:36:59.004792 29598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:58.992470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40334 (local address 127.10.131.62:37471)
0620 18:36:58.993701 (+ 1231us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:58.993711 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:58.993751 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:36:58.995432 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:58.995453 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:58.995891 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:58.996209 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:58.996809 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:58.998127 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:59.000137 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:59.001240 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:59.001392 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:59.002565 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:59.002572 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:59.002586 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:59.003076 (+ 490us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:59.003175 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:59.003202 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:59.003362 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:59.003620 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 18:36:59.003926 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":794,"thread_start_us":541,"threads_started":1}
I20260620 18:36:59.011142 29595 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb2abb31e70d45568a66963cb091c970 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2abb31e70d45568a66963cb091c970" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 37471 } }
I20260620 18:36:59.011494 29595 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb2abb31e70d45568a66963cb091c970 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:36:59.011610 29595 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb2abb31e70d45568a66963cb091c970 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb2abb31e70d45568a66963cb091c970, State: Initialized, Role: FOLLOWER
I20260620 18:36:59.012217 29595 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb2abb31e70d45568a66963cb091c970 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2abb31e70d45568a66963cb091c970" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 37471 } }
I20260620 18:36:59.012470 29595 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb2abb31e70d45568a66963cb091c970 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:36:59.012615 29595 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb2abb31e70d45568a66963cb091c970 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:36:59.012799 29595 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb2abb31e70d45568a66963cb091c970 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:36:59.015007 29595 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb2abb31e70d45568a66963cb091c970 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2abb31e70d45568a66963cb091c970" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 37471 } }
I20260620 18:36:59.015511 29595 leader_election.cc:302] T 00000000000000000000000000000000 P fb2abb31e70d45568a66963cb091c970 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb2abb31e70d45568a66963cb091c970; no voters:
I20260620 18:36:59.016170 29595 leader_election.cc:288] T 00000000000000000000000000000000 P fb2abb31e70d45568a66963cb091c970 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:36:59.016371 29600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb2abb31e70d45568a66963cb091c970 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:36:59.017462 29600 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb2abb31e70d45568a66963cb091c970 [term 1 LEADER]: Becoming Leader. State: Replica: fb2abb31e70d45568a66963cb091c970, State: Running, Role: LEADER
I20260620 18:36:59.018019 29600 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb2abb31e70d45568a66963cb091c970 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2abb31e70d45568a66963cb091c970" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 37471 } }
I20260620 18:36:59.018683 29595 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb2abb31e70d45568a66963cb091c970 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:36:59.021198 29601 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb2abb31e70d45568a66963cb091c970 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb2abb31e70d45568a66963cb091c970" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2abb31e70d45568a66963cb091c970" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 37471 } } }
I20260620 18:36:59.021234 29602 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb2abb31e70d45568a66963cb091c970 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb2abb31e70d45568a66963cb091c970. Latest consensus state: current_term: 1 leader_uuid: "fb2abb31e70d45568a66963cb091c970" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb2abb31e70d45568a66963cb091c970" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 37471 } } }
I20260620 18:36:59.021584 29601 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb2abb31e70d45568a66963cb091c970 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:59.021648 29602 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb2abb31e70d45568a66963cb091c970 [sys.catalog]: This master's current role is: LEADER
I20260620 18:36:59.025425 29606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:36:59.029767 29606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:36:59.038414 29606 catalog_manager.cc:1347] Generated new cluster ID: 712ad9e7edc946e8a86a213be8d988fe
I20260620 18:36:59.038532 29606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:36:59.050729 29606 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:36:59.052074 29606 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:36:59.058823 29606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb2abb31e70d45568a66963cb091c970: Generated new TSK 0
I20260620 18:36:59.059593 29606 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:36:59.068712 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.1:0
--local_ip_for_outbound_sockets=127.10.131.1
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:37471
--builtin_ntp_servers=127.10.131.20:46649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:36:59.399642 29619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:59.399899 29619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:59.399993 29619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:59.407212 29619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:59.407301 29619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:59.407505 29619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.1
I20260620 18:36:59.416633 29619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:46649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:37471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.1
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:36:59.417995 29619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:59.419762 29619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:59.432904 29625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:59.432890 29627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:59.433029 29624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:59.433705 29619 server_base.cc:1034] running on GCE node
I20260620 18:36:59.434242 29619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:59.435096 29619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:59.436328 29619 hybrid_clock.cc:648] HybridClock initialized: now 1781980619436274 us; error 49 us; skew 500 ppm
I20260620 18:36:59.436667 29619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:36:59.438757 29619 webserver.cc:460] Webserver started at http://127.10.131.1:35477/ using document root <none> and password file <none>
I20260620 18:36:59.439471 29619 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:59.439607 29619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:59.439921 29619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:59.443017 29619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-0/data/instance:
uuid: "412e9244e60d4f80a3ad49b96066acec"
format_stamp: "Formatted at 2026-06-20 18:36:59 on dist-test-slave-cbn3"
server_key: "b323df0cbc52f65c51fca31ca104bb7b"
server_key_iv: "cc7a99dcec5580567ad58849f0c3f071"
server_key_version: "encryptionkey@0"
I20260620 18:36:59.443825 29619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance:
uuid: "412e9244e60d4f80a3ad49b96066acec"
format_stamp: "Formatted at 2026-06-20 18:36:59 on dist-test-slave-cbn3"
server_key: "b323df0cbc52f65c51fca31ca104bb7b"
server_key_iv: "cc7a99dcec5580567ad58849f0c3f071"
server_key_version: "encryptionkey@0"
I20260620 18:36:59.449105 29619 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 18:36:59.452579 29633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:59.453903 29619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:36:59.454079 29619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-0/wal
uuid: "412e9244e60d4f80a3ad49b96066acec"
format_stamp: "Formatted at 2026-06-20 18:36:59 on dist-test-slave-cbn3"
server_key: "b323df0cbc52f65c51fca31ca104bb7b"
server_key_iv: "cc7a99dcec5580567ad58849f0c3f071"
server_key_version: "encryptionkey@0"
I20260620 18:36:59.454295 29619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:59.479391 29619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:59.480255 29619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:59.480571 29619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:59.481663 29619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:59.483057 29619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:59.483184 29619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:59.483317 29619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:59.483402 29619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:59.515925 29619 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.1:35437
I20260620 18:36:59.516036 29745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.1:35437 every 8 connection(s)
I20260620 18:36:59.517306 29619 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
I20260620 18:36:59.522488 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 29619
I20260620 18:36:59.522675 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance
I20260620 18:36:59.523216 10764 external_mini_cluster.cc:1428] Setting key 9909f5269678dc767bd689368b2e9151
I20260620 18:36:59.526034 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.2:0
--local_ip_for_outbound_sockets=127.10.131.2
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:37471
--builtin_ntp_servers=127.10.131.20:46649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:36:59.531270 29748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:59.519831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:36645 (local address 127.10.131.62:37471)
0620 18:36:59.520309 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:59.520313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:59.521378 (+ 1065us) server_negotiation.cc:407] Connection header received
0620 18:36:59.523688 (+ 2310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:59.523692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:59.523785 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:59.523906 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:59.525143 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:59.525845 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:59.528321 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:59.528778 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:59.528928 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:59.530517 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:59.530521 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:59.530525 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:59.530543 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:59.530586 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:59.530600 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:59.530669 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:59.530931 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 18:36:59.531008 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":254,"threads_started":1}
I20260620 18:36:59.532720 29749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:59.520356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:37471 (local address 127.10.131.1:36645)
0620 18:36:59.521226 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:59.521264 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:59.523401 (+ 2137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:59.524149 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:59.524160 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:59.524275 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:59.524950 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:59.524983 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:59.526010 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:59.526017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:59.528178 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:59.528196 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:59.529025 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:59.529033 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:59.529381 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:59.530134 (+ 753us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:36:59.530140 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:59.530199 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:36:59.530297 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:59.530700 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:59.530713 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:59.530724 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 18:36:59.530901 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 18:36:59.531325 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":260,"threads_started":1}
I20260620 18:36:59.534248 29746 heartbeater.cc:347] Connected to a master server at 127.10.131.62:37471
I20260620 18:36:59.534546 29746 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:59.535353 29746 heartbeater.cc:511] Master 127.10.131.62:37471 requested a full tablet report, sending...
I20260620 18:36:59.537525 29560 ts_manager.cc:194] Registered new tserver with Master: 412e9244e60d4f80a3ad49b96066acec (127.10.131.1:35437)
I20260620 18:36:59.539667 29560 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.131.1:36645
W20260620 18:36:59.846616 29750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:36:59.846871 29750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:36:59.846961 29750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:36:59.854177 29750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:36:59.854269 29750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:36:59.854369 29750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.2
I20260620 18:36:59.863055 29750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:46649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:37471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.2
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:36:59.864221 29750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:36:59.865924 29750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:36:59.879016 29758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:59.879123 29755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:36:59.879056 29756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:36:59.879778 29750 server_base.cc:1034] running on GCE node
I20260620 18:36:59.880409 29750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:36:59.881425 29750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:36:59.882640 29750 hybrid_clock.cc:648] HybridClock initialized: now 1781980619882586 us; error 45 us; skew 500 ppm
I20260620 18:36:59.882937 29750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:36:59.885071 29750 webserver.cc:460] Webserver started at http://127.10.131.2:39633/ using document root <none> and password file <none>
I20260620 18:36:59.885751 29750 fs_manager.cc:359] Metadata directory not provided
I20260620 18:36:59.885887 29750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:36:59.886199 29750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:36:59.889062 29750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-1/data/instance:
uuid: "6ea35066cfdd49ebb7a2bcf34da1e631"
format_stamp: "Formatted at 2026-06-20 18:36:59 on dist-test-slave-cbn3"
server_key: "94d5fd4afc02d7ca31b68729ed17b263"
server_key_iv: "9e0f3af94bc06f83b7ca684992b4de58"
server_key_version: "encryptionkey@0"
I20260620 18:36:59.889827 29750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance:
uuid: "6ea35066cfdd49ebb7a2bcf34da1e631"
format_stamp: "Formatted at 2026-06-20 18:36:59 on dist-test-slave-cbn3"
server_key: "94d5fd4afc02d7ca31b68729ed17b263"
server_key_iv: "9e0f3af94bc06f83b7ca684992b4de58"
server_key_version: "encryptionkey@0"
I20260620 18:36:59.894768 29750 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:36:59.897943 29764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:59.899152 29750 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 18:36:59.899331 29750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-1/wal
uuid: "6ea35066cfdd49ebb7a2bcf34da1e631"
format_stamp: "Formatted at 2026-06-20 18:36:59 on dist-test-slave-cbn3"
server_key: "94d5fd4afc02d7ca31b68729ed17b263"
server_key_iv: "9e0f3af94bc06f83b7ca684992b4de58"
server_key_version: "encryptionkey@0"
I20260620 18:36:59.899533 29750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:36:59.921926 29750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:36:59.922782 29750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:36:59.923094 29750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:36:59.924033 29750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:36:59.925511 29750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:36:59.925633 29750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:59.925767 29750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:36:59.925868 29750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:36:59.958370 29750 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.2:45755
I20260620 18:36:59.958415 29876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.2:45755 every 8 connection(s)
I20260620 18:36:59.959769 29750 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
I20260620 18:36:59.962829 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 29750
I20260620 18:36:59.963011 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance
I20260620 18:36:59.963601 10764 external_mini_cluster.cc:1428] Setting key beffd760d628fde01b9cad03c73d9849
I20260620 18:36:59.966665 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.3:0
--local_ip_for_outbound_sockets=127.10.131.3
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:37471
--builtin_ntp_servers=127.10.131.20:46649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:36:59.973474 29748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:59.962264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:39779 (local address 127.10.131.62:37471)
0620 18:36:59.962442 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 18:36:59.962446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:36:59.964039 (+ 1593us) server_negotiation.cc:407] Connection header received
0620 18:36:59.966187 (+ 2148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:36:59.966191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:36:59.966250 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:36:59.966325 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:36:59.967652 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:59.968328 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:59.970722 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:36:59.971117 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:36:59.971206 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:59.972763 (+ 1557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:36:59.972768 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:36:59.972771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:36:59.972788 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:36:59.972817 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:36:59.972828 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:36:59.972955 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 18:36:59.973187 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 18:36:59.973259 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:36:59.974879 29879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:36:59.962781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:37471 (local address 127.10.131.2:39779)
0620 18:36:59.963870 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0620 18:36:59.963924 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 18:36:59.965882 (+ 1958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:36:59.966563 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:36:59.966574 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:36:59.966689 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:36:59.967487 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:59.967514 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:59.968495 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:59.968503 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:59.970592 (+ 2089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:36:59.970610 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:36:59.971335 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:36:59.971343 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:36:59.971670 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:36:59.972433 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:36:59.972439 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:36:59.972489 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:36:59.972591 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:36:59.972963 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:36:59.972975 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:36:59.972986 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 18:36:59.973173 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 18:36:59.973568 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":335,"threads_started":1}
I20260620 18:36:59.976230 29877 heartbeater.cc:347] Connected to a master server at 127.10.131.62:37471
I20260620 18:36:59.976552 29877 heartbeater.cc:464] Registering TS with master...
I20260620 18:36:59.977367 29877 heartbeater.cc:511] Master 127.10.131.62:37471 requested a full tablet report, sending...
I20260620 18:36:59.978837 29560 ts_manager.cc:194] Registered new tserver with Master: 6ea35066cfdd49ebb7a2bcf34da1e631 (127.10.131.2:45755)
I20260620 18:36:59.979627 29560 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.131.2:39779
W20260620 18:37:00.298628 29880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:00.298887 29880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:00.299052 29880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:00.306324 29880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:00.306409 29880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:00.306588 29880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.3
I20260620 18:37:00.315358 29880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:46649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:37471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.3
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:37:00.316608 29880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:00.318338 29880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:00.331203 29886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:00.331560 29885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:00.331584 29888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:00.332481 29880 server_base.cc:1034] running on GCE node
I20260620 18:37:00.333233 29880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:00.334214 29880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:00.335460 29880 hybrid_clock.cc:648] HybridClock initialized: now 1781980620335400 us; error 43 us; skew 500 ppm
I20260620 18:37:00.335803 29880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:37:00.338166 29880 webserver.cc:460] Webserver started at http://127.10.131.3:37833/ using document root <none> and password file <none>
I20260620 18:37:00.338910 29880 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:00.339052 29880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:00.339389 29880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:00.342401 29880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-2/data/instance:
uuid: "7bfee6748dd8420d9c50c2b5089fd764"
format_stamp: "Formatted at 2026-06-20 18:37:00 on dist-test-slave-cbn3"
server_key: "a09df5b240a88c2693a9fa2a5f674f88"
server_key_iv: "aa8f2ac77e1f00aadc5d495e6ec442d8"
server_key_version: "encryptionkey@0"
I20260620 18:37:00.343271 29880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance:
uuid: "7bfee6748dd8420d9c50c2b5089fd764"
format_stamp: "Formatted at 2026-06-20 18:37:00 on dist-test-slave-cbn3"
server_key: "a09df5b240a88c2693a9fa2a5f674f88"
server_key_iv: "aa8f2ac77e1f00aadc5d495e6ec442d8"
server_key_version: "encryptionkey@0"
I20260620 18:37:00.348645 29880 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:37:00.352200 29894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:00.353498 29880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:37:00.353677 29880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-2/wal
uuid: "7bfee6748dd8420d9c50c2b5089fd764"
format_stamp: "Formatted at 2026-06-20 18:37:00 on dist-test-slave-cbn3"
server_key: "a09df5b240a88c2693a9fa2a5f674f88"
server_key_iv: "aa8f2ac77e1f00aadc5d495e6ec442d8"
server_key_version: "encryptionkey@0"
I20260620 18:37:00.353904 29880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:00.370239 29880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:00.371076 29880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:00.371419 29880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:00.372546 29880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:37:00.374272 29880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:37:00.374374 29880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:00.374478 29880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:37:00.374538 29880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:00.409293 29880 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.3:35523
I20260620 18:37:00.409330 30006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.3:35523 every 8 connection(s)
I20260620 18:37:00.410705 29880 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
I20260620 18:37:00.416239 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 29880
I20260620 18:37:00.416414 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance
I20260620 18:37:00.416961 10764 external_mini_cluster.cc:1428] Setting key 8ab7df986a82a60cb983d000754d65a2
I20260620 18:37:00.423555 29748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:00.413220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:57209 (local address 127.10.131.62:37471)
0620 18:37:00.413453 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:00.413460 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:00.414866 (+ 1406us) server_negotiation.cc:407] Connection header received
0620 18:37:00.416956 (+ 2090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:00.416961 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:00.417088 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:00.417193 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:00.418234 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:00.419165 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:00.420791 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:00.421279 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:00.421407 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:00.422749 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:00.422755 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:00.422761 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:00.422785 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:00.422815 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:00.422827 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:00.422919 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:00.423125 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 18:37:00.423231 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:37:00.424425 30009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:00.413832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:37471 (local address 127.10.131.3:57209)
0620 18:37:00.414695 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:00.414740 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:00.416585 (+ 1845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:00.417453 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:00.417463 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:00.417564 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:00.418074 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:00.418088 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:00.419338 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:00.419343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:00.420646 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:00.420657 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:00.421499 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:00.421506 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:00.421725 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:00.422404 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:37:00.422406 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:00.422458 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:37:00.422559 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:00.422918 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:00.422929 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:00.422939 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 18:37:00.423113 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 18:37:00.423472 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":275,"threads_started":1}
I20260620 18:37:00.425705 30007 heartbeater.cc:347] Connected to a master server at 127.10.131.62:37471
I20260620 18:37:00.425971 30007 heartbeater.cc:464] Registering TS with master...
I20260620 18:37:00.426661 30007 heartbeater.cc:511] Master 127.10.131.62:37471 requested a full tablet report, sending...
I20260620 18:37:00.427937 29560 ts_manager.cc:194] Registered new tserver with Master: 7bfee6748dd8420d9c50c2b5089fd764 (127.10.131.3:35523)
I20260620 18:37:00.428786 29560 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.131.3:57209
I20260620 18:37:00.432080 10764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:37:00.441943 29748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:00.435998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40348 (local address 127.10.131.62:37471)
0620 18:37:00.436239 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:00.436248 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:00.436607 (+ 359us) server_negotiation.cc:407] Connection header received
0620 18:37:00.436792 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:00.436798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:00.436908 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:00.437017 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:00.437508 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:00.438179 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:00.439542 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:00.439937 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:00.440072 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:00.441224 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:00.441237 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:00.441240 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:00.441257 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:00.441286 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:00.441297 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:00.441445 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:00.441635 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 18:37:00.441698 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:37:00.445513 29560 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:37:00.452176 29560 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:37:00.475028 30019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:00.464083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:35437 (local address 127.0.0.1:58368)
0620 18:37:00.465448 (+ 1365us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:00.465490 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:00.465743 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:00.466759 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:00.466766 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:00.466793 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:37:00.467190 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:00.467205 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:00.469193 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:00.469198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:00.471753 (+ 2555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:00.471768 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:00.474019 (+ 2251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:00.474025 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:00.474236 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:00.474294 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 18:37:00.474421 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 18:37:00.474626 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1148,"mutex_wait_us":47,"thread_start_us":1017,"threads_started":1}
I20260620 18:37:00.475437 30022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:00.465011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56492 (local address 127.10.131.3:35523)
0620 18:37:00.465646 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:00.465654 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:00.466689 (+ 1035us) server_negotiation.cc:407] Connection header received
0620 18:37:00.466784 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:00.466791 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:00.467032 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:00.467214 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:37:00.468112 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:00.469647 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:00.472175 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:00.473505 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:00.473619 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:00.474823 (+ 1204us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:00.474895 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 18:37:00.475083 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":355,"threads_started":1}
I20260620 18:37:00.475966 30020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:00.464278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58368 (local address 127.10.131.1:35437)
0620 18:37:00.464958 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:00.464967 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:00.465602 (+ 635us) server_negotiation.cc:407] Connection header received
0620 18:37:00.465994 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:00.466006 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:00.466263 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:00.466433 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:37:00.467366 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:00.469007 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:00.471987 (+ 2980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:00.473454 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:00.473592 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:00.473836 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:00.475402 (+ 1566us) server_negotiation.cc:299] Negotiation successful
0620 18:37:00.475616 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"spinlock_wait_cycles":22912,"thread_start_us":409,"threads_started":1}
I20260620 18:37:00.475972 30021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:00.464312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:35523 (local address 127.0.0.1:56492)
0620 18:37:00.466426 (+ 2114us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:00.466443 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:00.466553 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:00.467464 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:00.467469 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:00.467495 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:37:00.467865 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:00.467877 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:00.469806 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:00.469813 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:00.471957 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:00.471971 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:00.474225 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:00.474231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:00.474426 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:00.474455 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 18:37:00.474517 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 18:37:00.474625 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1960,"mutex_wait_us":1,"thread_start_us":1797,"threads_started":1}
I20260620 18:37:00.480823 29681 tablet_service.cc:1467] Processing CreateTablet for tablet 19a1feb7487c4d809a1a73fb31210464 (DEFAULT_TABLE table=test-table [id=16e0b40197304b4bb2b166dfa58748f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:37:00.482352 29681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19a1feb7487c4d809a1a73fb31210464. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:00.484081 30019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:00.466451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:45755 (local address 127.0.0.1:60794)
0620 18:37:00.476776 (+ 10325us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:00.476793 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:00.476917 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:00.477924 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:00.477929 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:00.477954 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:37:00.478119 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:00.478130 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:00.480095 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:00.480099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:00.481599 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:00.481612 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:00.483325 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:00.483331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:00.483512 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:00.483559 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 18:37:00.483655 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 18:37:00.483791 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10230,"thread_start_us":11134,"threads_started":1}
I20260620 18:37:00.484179 30024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:00.467082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60794 (local address 127.10.131.2:45755)
0620 18:37:00.477345 (+ 10263us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:00.477354 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:00.477381 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:37:00.477475 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:00.477482 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:00.477720 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:00.478290 (+ 570us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:37:00.478510 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:00.479941 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:00.481758 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:00.483159 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:00.483293 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:00.483537 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:00.483705 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 18:37:00.483865 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10115,"thread_start_us":10636,"threads_started":1}
I20260620 18:37:00.483964 29942 tablet_service.cc:1467] Processing CreateTablet for tablet 19a1feb7487c4d809a1a73fb31210464 (DEFAULT_TABLE table=test-table [id=16e0b40197304b4bb2b166dfa58748f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:37:00.485507 29942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19a1feb7487c4d809a1a73fb31210464. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:00.487888 29812 tablet_service.cc:1467] Processing CreateTablet for tablet 19a1feb7487c4d809a1a73fb31210464 (DEFAULT_TABLE table=test-table [id=16e0b40197304b4bb2b166dfa58748f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:37:00.489284 29812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19a1feb7487c4d809a1a73fb31210464. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:00.494469 30025 tablet_bootstrap.cc:492] T 19a1feb7487c4d809a1a73fb31210464 P 412e9244e60d4f80a3ad49b96066acec: Bootstrap starting.
I20260620 18:37:00.498464 30025 tablet_bootstrap.cc:654] T 19a1feb7487c4d809a1a73fb31210464 P 412e9244e60d4f80a3ad49b96066acec: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:00.499729 30025 log.cc:824] T 19a1feb7487c4d809a1a73fb31210464 P 412e9244e60d4f80a3ad49b96066acec: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:00.500022 30026 tablet_bootstrap.cc:492] T 19a1feb7487c4d809a1a73fb31210464 P 7bfee6748dd8420d9c50c2b5089fd764: Bootstrap starting.
I20260620 18:37:00.503710 30025 tablet_bootstrap.cc:492] T 19a1feb7487c4d809a1a73fb31210464 P 412e9244e60d4f80a3ad49b96066acec: No bootstrap required, opened a new log
I20260620 18:37:00.503894 30026 tablet_bootstrap.cc:654] T 19a1feb7487c4d809a1a73fb31210464 P 7bfee6748dd8420d9c50c2b5089fd764: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:00.503994 30028 tablet_bootstrap.cc:492] T 19a1feb7487c4d809a1a73fb31210464 P 6ea35066cfdd49ebb7a2bcf34da1e631: Bootstrap starting.
I20260620 18:37:00.504175 30025 ts_tablet_manager.cc:1397] T 19a1feb7487c4d809a1a73fb31210464 P 412e9244e60d4f80a3ad49b96066acec: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 18:37:00.506292 30026 log.cc:824] T 19a1feb7487c4d809a1a73fb31210464 P 7bfee6748dd8420d9c50c2b5089fd764: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:00.507637 30028 tablet_bootstrap.cc:654] T 19a1feb7487c4d809a1a73fb31210464 P 6ea35066cfdd49ebb7a2bcf34da1e631: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:00.508627 30028 log.cc:824] T 19a1feb7487c4d809a1a73fb31210464 P 6ea35066cfdd49ebb7a2bcf34da1e631: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:00.509509 30026 tablet_bootstrap.cc:492] T 19a1feb7487c4d809a1a73fb31210464 P 7bfee6748dd8420d9c50c2b5089fd764: No bootstrap required, opened a new log
I20260620 18:37:00.509785 30026 ts_tablet_manager.cc:1397] T 19a1feb7487c4d809a1a73fb31210464 P 7bfee6748dd8420d9c50c2b5089fd764: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 18:37:00.511057 30028 tablet_bootstrap.cc:492] T 19a1feb7487c4d809a1a73fb31210464 P 6ea35066cfdd49ebb7a2bcf34da1e631: No bootstrap required, opened a new log
I20260620 18:37:00.511312 30028 ts_tablet_manager.cc:1397] T 19a1feb7487c4d809a1a73fb31210464 P 6ea35066cfdd49ebb7a2bcf34da1e631: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260620 18:37:00.514650 30025 raft_consensus.cc:348] T 19a1feb7487c4d809a1a73fb31210464 P 412e9244e60d4f80a3ad49b96066acec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e9244e60d4f80a3ad49b96066acec" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35437 } } peers { permanent_uuid: "7bfee6748dd8420d9c50c2b5089fd764" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35523 } } peers { permanent_uuid: "6ea35066cfdd49ebb7a2bcf34da1e631" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 45755 } }
I20260620 18:37:00.515113 30025 raft_consensus.cc:374] T 19a1feb7487c4d809a1a73fb31210464 P 412e9244e60d4f80a3ad49b96066acec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:00.515285 30025 raft_consensus.cc:729] T 19a1feb7487c4d809a1a73fb31210464 P 412e9244e60d4f80a3ad49b96066acec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412e9244e60d4f80a3ad49b96066acec, State: Initialized, Role: FOLLOWER
I20260620 18:37:00.516161 30025 consensus_queue.cc:260] T 19a1feb7487c4d809a1a73fb31210464 P 412e9244e60d4f80a3ad49b96066acec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e9244e60d4f80a3ad49b96066acec" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35437 } } peers { permanent_uuid: "7bfee6748dd8420d9c50c2b5089fd764" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35523 } } peers { permanent_uuid: "6ea35066cfdd49ebb7a2bcf34da1e631" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 45755 } }
I20260620 18:37:00.519708 30028 raft_consensus.cc:348] T 19a1feb7487c4d809a1a73fb31210464 P 6ea35066cfdd49ebb7a2bcf34da1e631 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e9244e60d4f80a3ad49b96066acec" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35437 } } peers { permanent_uuid: "7bfee6748dd8420d9c50c2b5089fd764" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35523 } } peers { permanent_uuid: "6ea35066cfdd49ebb7a2bcf34da1e631" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 45755 } }
I20260620 18:37:00.520263 30028 raft_consensus.cc:374] T 19a1feb7487c4d809a1a73fb31210464 P 6ea35066cfdd49ebb7a2bcf34da1e631 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:00.520390 30028 raft_consensus.cc:729] T 19a1feb7487c4d809a1a73fb31210464 P 6ea35066cfdd49ebb7a2bcf34da1e631 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea35066cfdd49ebb7a2bcf34da1e631, State: Initialized, Role: FOLLOWER
I20260620 18:37:00.521126 30028 consensus_queue.cc:260] T 19a1feb7487c4d809a1a73fb31210464 P 6ea35066cfdd49ebb7a2bcf34da1e631 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e9244e60d4f80a3ad49b96066acec" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35437 } } peers { permanent_uuid: "7bfee6748dd8420d9c50c2b5089fd764" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35523 } } peers { permanent_uuid: "6ea35066cfdd49ebb7a2bcf34da1e631" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 45755 } }
I20260620 18:37:00.523409 29746 heartbeater.cc:503] Master 127.10.131.62:37471 was elected leader, sending a full tablet report...
I20260620 18:37:00.524436 30025 ts_tablet_manager.cc:1428] T 19a1feb7487c4d809a1a73fb31210464 P 412e9244e60d4f80a3ad49b96066acec: Time spent starting tablet: real 0.020s user 0.014s sys 0.004s
I20260620 18:37:00.525877 30028 ts_tablet_manager.cc:1428] T 19a1feb7487c4d809a1a73fb31210464 P 6ea35066cfdd49ebb7a2bcf34da1e631: Time spent starting tablet: real 0.014s user 0.005s sys 0.009s
I20260620 18:37:00.526129 29877 heartbeater.cc:503] Master 127.10.131.62:37471 was elected leader, sending a full tablet report...
I20260620 18:37:00.527778 30026 raft_consensus.cc:348] T 19a1feb7487c4d809a1a73fb31210464 P 7bfee6748dd8420d9c50c2b5089fd764 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e9244e60d4f80a3ad49b96066acec" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35437 } } peers { permanent_uuid: "7bfee6748dd8420d9c50c2b5089fd764" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35523 } } peers { permanent_uuid: "6ea35066cfdd49ebb7a2bcf34da1e631" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 45755 } }
I20260620 18:37:00.528208 30026 raft_consensus.cc:374] T 19a1feb7487c4d809a1a73fb31210464 P 7bfee6748dd8420d9c50c2b5089fd764 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:00.528326 30026 raft_consensus.cc:729] T 19a1feb7487c4d809a1a73fb31210464 P 7bfee6748dd8420d9c50c2b5089fd764 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bfee6748dd8420d9c50c2b5089fd764, State: Initialized, Role: FOLLOWER
I20260620 18:37:00.528970 30026 consensus_queue.cc:260] T 19a1feb7487c4d809a1a73fb31210464 P 7bfee6748dd8420d9c50c2b5089fd764 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e9244e60d4f80a3ad49b96066acec" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35437 } } peers { permanent_uuid: "7bfee6748dd8420d9c50c2b5089fd764" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35523 } } peers { permanent_uuid: "6ea35066cfdd49ebb7a2bcf34da1e631" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 45755 } }
I20260620 18:37:00.531757 30007 heartbeater.cc:503] Master 127.10.131.62:37471 was elected leader, sending a full tablet report...
I20260620 18:37:00.532227 30026 ts_tablet_manager.cc:1428] T 19a1feb7487c4d809a1a73fb31210464 P 7bfee6748dd8420d9c50c2b5089fd764: Time spent starting tablet: real 0.022s user 0.016s sys 0.000s
W20260620 18:37:00.662472 30008 tablet.cc:2369] T 19a1feb7487c4d809a1a73fb31210464 P 7bfee6748dd8420d9c50c2b5089fd764: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:37:00.711927 29878 tablet.cc:2369] T 19a1feb7487c4d809a1a73fb31210464 P 6ea35066cfdd49ebb7a2bcf34da1e631: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:37:00.735819 30032 raft_consensus.cc:482] T 19a1feb7487c4d809a1a73fb31210464 P 6ea35066cfdd49ebb7a2bcf34da1e631 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:37:00.736125 30032 raft_consensus.cc:504] T 19a1feb7487c4d809a1a73fb31210464 P 6ea35066cfdd49ebb7a2bcf34da1e631 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e9244e60d4f80a3ad49b96066acec" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35437 } } peers { permanent_uuid: "7bfee6748dd8420d9c50c2b5089fd764" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35523 } } peers { permanent_uuid: "6ea35066cfdd49ebb7a2bcf34da1e631" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 45755 } }
I20260620 18:37:00.737891 30032 leader_election.cc:288] T 19a1feb7487c4d809a1a73fb31210464 P 6ea35066cfdd49ebb7a2bcf34da1e631 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 412e9244e60d4f80a3ad49b96066acec (127.10.131.1:35437), 7bfee6748dd8420d9c50c2b5089fd764 (127.10.131.3:35523)
I20260620 18:37:00.746479 30022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:00.738398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:55627 (local address 127.10.131.3:35523)
0620 18:37:00.738645 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:00.738651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:00.739828 (+ 1177us) server_negotiation.cc:407] Connection header received
0620 18:37:00.740059 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:00.740063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:00.740120 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:00.740190 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:37:00.740746 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:00.741746 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:00.744138 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:00.745416 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:00.745590 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:00.745659 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:00.746117 (+ 458us) server_negotiation.cc:299] Negotiation successful
0620 18:37:00.746233 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260620 18:37:00.746482 30020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:00.738437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:54417 (local address 127.10.131.1:35437)
0620 18:37:00.738645 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:00.738651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:00.739272 (+ 621us) server_negotiation.cc:407] Connection header received
0620 18:37:00.739561 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:00.739567 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:00.739661 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:00.739740 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:37:00.740356 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:00.741417 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:00.744143 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:00.745411 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:00.745592 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:00.745659 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:00.746108 (+ 449us) server_negotiation.cc:299] Negotiation successful
0620 18:37:00.746232 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 18:37:00.746613 30034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:00.738417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:35437 (local address 127.10.131.2:54417)
0620 18:37:00.739127 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:00.739153 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:00.739333 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:00.739798 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:00.739803 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:00.739830 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:37:00.740208 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:00.740222 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:00.741547 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:00.741551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:00.743915 (+ 2364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:00.743936 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:00.745682 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:00.745687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:00.745889 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:00.745964 (+ 75us) client_negotiation.cc:769] Sending connection context
0620 18:37:00.746061 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 18:37:00.746215 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"mutex_wait_us":85,"thread_start_us":420,"threads_started":1}
I20260620 18:37:00.746858 30035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:00.738454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:35523 (local address 127.10.131.2:55627)
0620 18:37:00.739695 (+ 1241us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:00.739707 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:00.739912 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:00.740237 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:00.740240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:00.740270 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:37:00.740580 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:00.740587 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:00.741923 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:00.741929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:00.743971 (+ 2042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:00.743986 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:00.745677 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:00.745684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:00.745891 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:00.745966 (+ 75us) client_negotiation.cc:769] Sending connection context
0620 18:37:00.746061 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 18:37:00.746215 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1154,"mutex_wait_us":80,"spinlock_wait_cycles":22912,"thread_start_us":971,"threads_started":1}
I20260620 18:37:00.747611 29701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19a1feb7487c4d809a1a73fb31210464" candidate_uuid: "6ea35066cfdd49ebb7a2bcf34da1e631" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412e9244e60d4f80a3ad49b96066acec" is_pre_election: true
I20260620 18:37:00.747689 29962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19a1feb7487c4d809a1a73fb31210464" candidate_uuid: "6ea35066cfdd49ebb7a2bcf34da1e631" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bfee6748dd8420d9c50c2b5089fd764" is_pre_election: true
I20260620 18:37:00.748104 29701 raft_consensus.cc:2454] T 19a1feb7487c4d809a1a73fb31210464 P 412e9244e60d4f80a3ad49b96066acec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea35066cfdd49ebb7a2bcf34da1e631 in term 0.
I20260620 18:37:00.748114 29962 raft_consensus.cc:2454] T 19a1feb7487c4d809a1a73fb31210464 P 7bfee6748dd8420d9c50c2b5089fd764 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea35066cfdd49ebb7a2bcf34da1e631 in term 0.
I20260620 18:37:00.748728 29766 leader_election.cc:302] T 19a1feb7487c4d809a1a73fb31210464 P 6ea35066cfdd49ebb7a2bcf34da1e631 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 412e9244e60d4f80a3ad49b96066acec, 6ea35066cfdd49ebb7a2bcf34da1e631; no voters:
I20260620 18:37:00.749131 30032 raft_consensus.cc:2781] T 19a1feb7487c4d809a1a73fb31210464 P 6ea35066cfdd49ebb7a2bcf34da1e631 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:37:00.749274 30032 raft_consensus.cc:482] T 19a1feb7487c4d809a1a73fb31210464 P 6ea35066cfdd49ebb7a2bcf34da1e631 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:37:00.749370 30032 raft_consensus.cc:3037] T 19a1feb7487c4d809a1a73fb31210464 P 6ea35066cfdd49ebb7a2bcf34da1e631 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:00.751426 30032 raft_consensus.cc:504] T 19a1feb7487c4d809a1a73fb31210464 P 6ea35066cfdd49ebb7a2bcf34da1e631 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e9244e60d4f80a3ad49b96066acec" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35437 } } peers { permanent_uuid: "7bfee6748dd8420d9c50c2b5089fd764" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35523 } } peers { permanent_uuid: "6ea35066cfdd49ebb7a2bcf34da1e631" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 45755 } }
I20260620 18:37:00.751994 30032 leader_election.cc:288] T 19a1feb7487c4d809a1a73fb31210464 P 6ea35066cfdd49ebb7a2bcf34da1e631 [CANDIDATE]: Term 1 election: Requested vote from peers 412e9244e60d4f80a3ad49b96066acec (127.10.131.1:35437), 7bfee6748dd8420d9c50c2b5089fd764 (127.10.131.3:35523)
I20260620 18:37:00.752406 29701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19a1feb7487c4d809a1a73fb31210464" candidate_uuid: "6ea35066cfdd49ebb7a2bcf34da1e631" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412e9244e60d4f80a3ad49b96066acec"
I20260620 18:37:00.752503 29962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19a1feb7487c4d809a1a73fb31210464" candidate_uuid: "6ea35066cfdd49ebb7a2bcf34da1e631" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bfee6748dd8420d9c50c2b5089fd764"
I20260620 18:37:00.752619 29701 raft_consensus.cc:3037] T 19a1feb7487c4d809a1a73fb31210464 P 412e9244e60d4f80a3ad49b96066acec [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:00.752696 29962 raft_consensus.cc:3037] T 19a1feb7487c4d809a1a73fb31210464 P 7bfee6748dd8420d9c50c2b5089fd764 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:00.755786 29701 raft_consensus.cc:2454] T 19a1feb7487c4d809a1a73fb31210464 P 412e9244e60d4f80a3ad49b96066acec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea35066cfdd49ebb7a2bcf34da1e631 in term 1.
I20260620 18:37:00.755786 29962 raft_consensus.cc:2454] T 19a1feb7487c4d809a1a73fb31210464 P 7bfee6748dd8420d9c50c2b5089fd764 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea35066cfdd49ebb7a2bcf34da1e631 in term 1.
I20260620 18:37:00.756211 29768 leader_election.cc:302] T 19a1feb7487c4d809a1a73fb31210464 P 6ea35066cfdd49ebb7a2bcf34da1e631 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ea35066cfdd49ebb7a2bcf34da1e631, 7bfee6748dd8420d9c50c2b5089fd764; no voters:
I20260620 18:37:00.756460 30032 raft_consensus.cc:2781] T 19a1feb7487c4d809a1a73fb31210464 P 6ea35066cfdd49ebb7a2bcf34da1e631 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:37:00.756949 30032 raft_consensus.cc:686] T 19a1feb7487c4d809a1a73fb31210464 P 6ea35066cfdd49ebb7a2bcf34da1e631 [term 1 LEADER]: Becoming Leader. State: Replica: 6ea35066cfdd49ebb7a2bcf34da1e631, State: Running, Role: LEADER
I20260620 18:37:00.757485 30032 consensus_queue.cc:237] T 19a1feb7487c4d809a1a73fb31210464 P 6ea35066cfdd49ebb7a2bcf34da1e631 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e9244e60d4f80a3ad49b96066acec" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35437 } } peers { permanent_uuid: "7bfee6748dd8420d9c50c2b5089fd764" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35523 } } peers { permanent_uuid: "6ea35066cfdd49ebb7a2bcf34da1e631" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 45755 } }
I20260620 18:37:00.761850 29560 catalog_manager.cc:5495] T 19a1feb7487c4d809a1a73fb31210464 P 6ea35066cfdd49ebb7a2bcf34da1e631 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ea35066cfdd49ebb7a2bcf34da1e631 (127.10.131.2). New cstate: current_term: 1 leader_uuid: "6ea35066cfdd49ebb7a2bcf34da1e631" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e9244e60d4f80a3ad49b96066acec" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bfee6748dd8420d9c50c2b5089fd764" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ea35066cfdd49ebb7a2bcf34da1e631" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 45755 } health_report { overall_health: HEALTHY } } }
W20260620 18:37:00.769757 29747 tablet.cc:2369] T 19a1feb7487c4d809a1a73fb31210464 P 412e9244e60d4f80a3ad49b96066acec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:37:00.843645 30024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:00.838605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60810 (local address 127.10.131.2:45755)
0620 18:37:00.838818 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:00.838823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:00.838841 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:37:00.838991 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:00.838994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:00.839110 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:00.839235 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:37:00.839570 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:00.840239 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:00.841615 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:00.842022 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:00.842110 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:00.842599 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:37:00.843063 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:37:00.843186 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:00.843305 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 18:37:00.843364 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 18:37:00.851450 29962 raft_consensus.cc:1261] T 19a1feb7487c4d809a1a73fb31210464 P 7bfee6748dd8420d9c50c2b5089fd764 [term 1 FOLLOWER]: Refusing update from remote peer 6ea35066cfdd49ebb7a2bcf34da1e631: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:37:00.851471 29701 raft_consensus.cc:1261] T 19a1feb7487c4d809a1a73fb31210464 P 412e9244e60d4f80a3ad49b96066acec [term 1 FOLLOWER]: Refusing update from remote peer 6ea35066cfdd49ebb7a2bcf34da1e631: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:37:00.852593 30036 consensus_queue.cc:1035] T 19a1feb7487c4d809a1a73fb31210464 P 6ea35066cfdd49ebb7a2bcf34da1e631 [LEADER]: Connected to new peer: Peer: permanent_uuid: "412e9244e60d4f80a3ad49b96066acec" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 35437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:37:00.852921 30032 consensus_queue.cc:1035] T 19a1feb7487c4d809a1a73fb31210464 P 6ea35066cfdd49ebb7a2bcf34da1e631 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bfee6748dd8420d9c50c2b5089fd764" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 35523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:37:00.865003 30039 mvcc.cc:205] Tried to move back new op lower bound from 7298992622997798912 to 7298992622626729984. Current Snapshot: MvccSnapshot[applied={T|T < 7298992622997798912}]
I20260620 18:37:00.889966 29559 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:37:00.890273 29559 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:37:00.894205 29559 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb2abb31e70d45568a66963cb091c970: Sending DeleteTablet for 3 replicas of tablet 19a1feb7487c4d809a1a73fb31210464
I20260620 18:37:00.895320 29681 tablet_service.cc:1514] Processing DeleteTablet for tablet 19a1feb7487c4d809a1a73fb31210464 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:37:00 UTC) from {username='slave'} at 127.0.0.1:58368
I20260620 18:37:00.895442 29942 tablet_service.cc:1514] Processing DeleteTablet for tablet 19a1feb7487c4d809a1a73fb31210464 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:37:00 UTC) from {username='slave'} at 127.0.0.1:56492
I20260620 18:37:00.895630 29812 tablet_service.cc:1514] Processing DeleteTablet for tablet 19a1feb7487c4d809a1a73fb31210464 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:37:00 UTC) from {username='slave'} at 127.0.0.1:60794
I20260620 18:37:00.896476 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 29619
I20260620 18:37:00.907150 30051 tablet_replica.cc:330] T 19a1feb7487c4d809a1a73fb31210464 P 6ea35066cfdd49ebb7a2bcf34da1e631: stopping tablet replica
I20260620 18:37:00.907866 30051 raft_consensus.cc:2229] T 19a1feb7487c4d809a1a73fb31210464 P 6ea35066cfdd49ebb7a2bcf34da1e631 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:37:00.908878 30051 raft_consensus.cc:2258] T 19a1feb7487c4d809a1a73fb31210464 P 6ea35066cfdd49ebb7a2bcf34da1e631 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:37:00.909569 30049 tablet_replica.cc:330] T 19a1feb7487c4d809a1a73fb31210464 P 412e9244e60d4f80a3ad49b96066acec: stopping tablet replica
I20260620 18:37:00.910027 30049 raft_consensus.cc:2229] T 19a1feb7487c4d809a1a73fb31210464 P 412e9244e60d4f80a3ad49b96066acec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:37:00.910646 30050 tablet_replica.cc:330] T 19a1feb7487c4d809a1a73fb31210464 P 7bfee6748dd8420d9c50c2b5089fd764: stopping tablet replica
I20260620 18:37:00.911022 30050 raft_consensus.cc:2229] T 19a1feb7487c4d809a1a73fb31210464 P 7bfee6748dd8420d9c50c2b5089fd764 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:37:00.911554 30050 raft_consensus.cc:2258] T 19a1feb7487c4d809a1a73fb31210464 P 7bfee6748dd8420d9c50c2b5089fd764 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:37:00.912217 30049 raft_consensus.cc:2258] T 19a1feb7487c4d809a1a73fb31210464 P 412e9244e60d4f80a3ad49b96066acec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:37:00.913511 30051 ts_tablet_manager.cc:1905] T 19a1feb7487c4d809a1a73fb31210464 P 6ea35066cfdd49ebb7a2bcf34da1e631: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:37:00.914345 30049 ts_tablet_manager.cc:1905] T 19a1feb7487c4d809a1a73fb31210464 P 412e9244e60d4f80a3ad49b96066acec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:37:00.918292 30020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:00.899184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58380 (local address 127.10.131.1:35437)
0620 18:37:00.899403 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:00.899412 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:00.909665 (+ 10253us) server_negotiation.cc:407] Connection header received
0620 18:37:00.910157 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:00.910162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:00.910260 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:00.910342 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:00.911063 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:00.912209 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:00.914079 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:00.914741 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:00.914851 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:00.916520 (+ 1669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:00.916530 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:00.916547 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:00.917324 (+ 777us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:00.917391 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:00.917414 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:00.917493 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:00.917736 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 18:37:00.917850 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:37:00.919438 29741 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:00.920786 30050 ts_tablet_manager.cc:1905] T 19a1feb7487c4d809a1a73fb31210464 P 7bfee6748dd8420d9c50c2b5089fd764: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:37:00.923439 30051 ts_tablet_manager.cc:1918] T 19a1feb7487c4d809a1a73fb31210464 P 6ea35066cfdd49ebb7a2bcf34da1e631: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:37:00.923887 30051 log.cc:1196] T 19a1feb7487c4d809a1a73fb31210464 P 6ea35066cfdd49ebb7a2bcf34da1e631: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-1/wal/wals/19a1feb7487c4d809a1a73fb31210464
I20260620 18:37:00.924533 30051 ts_tablet_manager.cc:1939] T 19a1feb7487c4d809a1a73fb31210464 P 6ea35066cfdd49ebb7a2bcf34da1e631: Deleting consensus metadata
I20260620 18:37:00.926870 29547 catalog_manager.cc:4841] TS 6ea35066cfdd49ebb7a2bcf34da1e631 (127.10.131.2:45755): tablet 19a1feb7487c4d809a1a73fb31210464 (table test-table [id=16e0b40197304b4bb2b166dfa58748f6]) successfully deleted
I20260620 18:37:00.928793 30050 ts_tablet_manager.cc:1918] T 19a1feb7487c4d809a1a73fb31210464 P 7bfee6748dd8420d9c50c2b5089fd764: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:37:00.929047 30050 log.cc:1196] T 19a1feb7487c4d809a1a73fb31210464 P 7bfee6748dd8420d9c50c2b5089fd764: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781980314095189-10764-0/minicluster-data/ts-2/wal/wals/19a1feb7487c4d809a1a73fb31210464
I20260620 18:37:00.929606 30050 ts_tablet_manager.cc:1939] T 19a1feb7487c4d809a1a73fb31210464 P 7bfee6748dd8420d9c50c2b5089fd764: Deleting consensus metadata
I20260620 18:37:00.931638 29547 catalog_manager.cc:4841] TS 7bfee6748dd8420d9c50c2b5089fd764 (127.10.131.3:35523): tablet 19a1feb7487c4d809a1a73fb31210464 (table test-table [id=16e0b40197304b4bb2b166dfa58748f6]) successfully deleted
I20260620 18:37:01.034711 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 29619
W20260620 18:37:01.058946 29545 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.131.1:35437 (error 108)
I20260620 18:37:01.060207 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 29750
I20260620 18:37:01.068297 30055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:01.060174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:35437 (local address 127.0.0.1:58388)
0620 18:37:01.067911 (+ 7737us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:01.068073 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:35437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7346,"thread_start_us":7246,"threads_started":1}
W20260620 18:37:01.068626 29545 catalog_manager.cc:4584] TS 412e9244e60d4f80a3ad49b96066acec (127.10.131.1:35437): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 19a1feb7487c4d809a1a73fb31210464: Network error: Client connection negotiation failed: client connection to 127.10.131.1:35437: connect: Connection refused (error 111)
I20260620 18:37:01.069798 30024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:01.061142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60820 (local address 127.10.131.2:45755)
0620 18:37:01.061378 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:01.061387 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:01.061412 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:37:01.061596 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:01.061605 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:01.061742 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:01.061882 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:01.062350 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:01.063930 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:01.065931 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:01.066894 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:01.067057 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:01.068160 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:01.068171 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:01.068193 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:01.069061 (+ 868us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:01.069110 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:01.069134 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:01.069212 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:01.069383 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 18:37:01.069460 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 18:37:01.070539 29872 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:01.122148 30055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:01.121666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:35437 (local address 127.0.0.1:58400)
0620 18:37:01.121905 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:01.122001 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:35437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 18:37:01.123183 30055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:01.122813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:35437 (local address 127.0.0.1:58408)
0620 18:37:01.122959 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:01.123040 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:35437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 18:37:01.180459 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 29750
I20260620 18:37:01.183585 30055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:01.183145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:35437 (local address 127.0.0.1:58410)
0620 18:37:01.183363 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:01.183440 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:35437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 18:37:01.184549 30055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:01.184204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:35437 (local address 127.0.0.1:58414)
0620 18:37:01.184360 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:01.184425 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:35437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 18:37:01.199429 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 29880
I20260620 18:37:01.206616 30022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:01.200156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56498 (local address 127.10.131.3:35523)
0620 18:37:01.200357 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:01.200366 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:01.200387 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:37:01.200640 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:01.200645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:01.200727 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:01.200840 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:01.201412 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:01.202187 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:01.203551 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:01.203994 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:01.204139 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:01.205249 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:01.205257 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:01.205271 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:01.205839 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:01.205896 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:01.205916 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:01.206044 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:01.206178 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 18:37:01.206286 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 18:37:01.207281 30002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:01.265841 30055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:01.265282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:35437 (local address 127.0.0.1:58418)
0620 18:37:01.265550 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:01.265677 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:35437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 18:37:01.266957 30055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:01.266545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:35437 (local address 127.0.0.1:58420)
0620 18:37:01.266742 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:01.266832 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:35437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260620 18:37:01.294888 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 29880
I20260620 18:37:01.311633 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 29529
I20260620 18:37:01.312727 29590 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:01.422689 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 29529
2026-06-20T18:37:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2959 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:37:01 dist-test-slave-cbn3 krb5kdc[30061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:37:01 dist-test-slave-cbn3 krb5kdc[30061](info): set up 2 sockets
Jun 20 18:37:01 dist-test-slave-cbn3 krb5kdc[30061](info): commencing operation
krb5kdc: starting...
W20260620 18:37:02.561199 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:37:02 dist-test-slave-cbn3 krb5kdc[30061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:37:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:37:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.131.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
I20260620 18:37:02.808192 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:36063
--webserver_interface=127.10.131.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:42211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:36063
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:03.136441 30074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:03.136785 30074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:03.136986 30074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:03.144327 30074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:37:03.144480 30074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:03.144588 30074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:03.144706 30074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:37:03.144794 30074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:37:03.154822 30074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:42211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:36063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:36063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:37:03.156183 30074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:03.157574 30074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:03.169587 30079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:03.169752 30082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:03.169598 30080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:03.170256 30074 server_base.cc:1034] running on GCE node
I20260620 18:37:03.171113 30074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:03.172544 30074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:03.173774 30074 hybrid_clock.cc:648] HybridClock initialized: now 1781980623173724 us; error 60 us; skew 500 ppm
I20260620 18:37:03.174144 30074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:03 dist-test-slave-cbn3 krb5kdc[30061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980623, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:03.177400 30074 init.cc:377] Logged in from keytab as kudu/127.10.131.62@KRBTEST.COM (short username kudu)
I20260620 18:37:03.178915 30074 webserver.cc:460] Webserver started at http://127.10.131.62:42855/ using document root <none> and password file <none>
I20260620 18:37:03.179603 30074 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:03.179752 30074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:03.180064 30074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:03.183070 30074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "8a2b9515eba149bcbaf3d4e7f5b4c994"
format_stamp: "Formatted at 2026-06-20 18:37:03 on dist-test-slave-cbn3"
server_key: "c2901e55f6d119829369ece301e6a82d"
server_key_iv: "e36aadcc6bf0e281a71785ccc5246375"
server_key_version: "encryptionkey@0"
I20260620 18:37:03.183811 30074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "8a2b9515eba149bcbaf3d4e7f5b4c994"
format_stamp: "Formatted at 2026-06-20 18:37:03 on dist-test-slave-cbn3"
server_key: "c2901e55f6d119829369ece301e6a82d"
server_key_iv: "e36aadcc6bf0e281a71785ccc5246375"
server_key_version: "encryptionkey@0"
I20260620 18:37:03.188999 30074 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 18:37:03.192400 30089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:03.193743 30074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:37:03.193943 30074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "8a2b9515eba149bcbaf3d4e7f5b4c994"
format_stamp: "Formatted at 2026-06-20 18:37:03 on dist-test-slave-cbn3"
server_key: "c2901e55f6d119829369ece301e6a82d"
server_key_iv: "e36aadcc6bf0e281a71785ccc5246375"
server_key_version: "encryptionkey@0"
I20260620 18:37:03.194146 30074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:03.221279 30074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:03.225790 30074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:03.226096 30074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:03.244397 30074 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:36063
I20260620 18:37:03.244431 30140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:36063 every 8 connection(s)
I20260620 18:37:03.245863 30074 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:37:03.249751 30141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:03.251884 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 30074
I20260620 18:37:03.252108 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:37:03.252702 10764 external_mini_cluster.cc:1428] Setting key e8ba347fdcfb33a8b943c6c92bcc8207
I20260620 18:37:03.263267 30141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a2b9515eba149bcbaf3d4e7f5b4c994: Bootstrap starting.
I20260620 18:37:03.266472 30141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a2b9515eba149bcbaf3d4e7f5b4c994: Neither blocks nor log segments found. Creating new log.
Jun 20 18:37:03 dist-test-slave-cbn3 krb5kdc[30061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:37:03.267774 30141 log.cc:824] T 00000000000000000000000000000000 P 8a2b9515eba149bcbaf3d4e7f5b4c994: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:03.270988 30141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a2b9515eba149bcbaf3d4e7f5b4c994: No bootstrap required, opened a new log
I20260620 18:37:03.275657 30144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:03.254916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42320 (local address 127.10.131.62:36063)
0620 18:37:03.255792 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:03.255805 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:03.255853 (+ 48us) server_negotiation.cc:407] Connection header received
0620 18:37:03.257573 (+ 1720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:03.257601 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:03.258017 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:03.258390 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:03.259914 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:03.261334 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:03.262715 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:03.263422 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:03.263651 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:03.268190 (+ 4539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:03.268198 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:03.268213 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:03.268294 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:03.271590 (+ 3296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:03.272480 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:03.272486 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:03.272492 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:03.272649 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:03.273087 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:03.273090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:03.273092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:03.273596 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:03.273808 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:03.274279 (+ 471us) server_negotiation.cc:299] Negotiation successful
0620 18:37:03.274600 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":400,"threads_started":1}
I20260620 18:37:03.280120 30141 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a2b9515eba149bcbaf3d4e7f5b4c994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a2b9515eba149bcbaf3d4e7f5b4c994" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 36063 } }
I20260620 18:37:03.280493 30141 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a2b9515eba149bcbaf3d4e7f5b4c994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:03.280609 30141 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a2b9515eba149bcbaf3d4e7f5b4c994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2b9515eba149bcbaf3d4e7f5b4c994, State: Initialized, Role: FOLLOWER
I20260620 18:37:03.281311 30141 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a2b9515eba149bcbaf3d4e7f5b4c994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a2b9515eba149bcbaf3d4e7f5b4c994" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 36063 } }
I20260620 18:37:03.281574 30141 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a2b9515eba149bcbaf3d4e7f5b4c994 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:37:03.281700 30141 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a2b9515eba149bcbaf3d4e7f5b4c994 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:37:03.281874 30141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a2b9515eba149bcbaf3d4e7f5b4c994 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:03.284143 30141 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a2b9515eba149bcbaf3d4e7f5b4c994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a2b9515eba149bcbaf3d4e7f5b4c994" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 36063 } }
I20260620 18:37:03.284613 30141 leader_election.cc:302] T 00000000000000000000000000000000 P 8a2b9515eba149bcbaf3d4e7f5b4c994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a2b9515eba149bcbaf3d4e7f5b4c994; no voters:
I20260620 18:37:03.285197 30141 leader_election.cc:288] T 00000000000000000000000000000000 P 8a2b9515eba149bcbaf3d4e7f5b4c994 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:37:03.285344 30146 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a2b9515eba149bcbaf3d4e7f5b4c994 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:37:03.286068 30146 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a2b9515eba149bcbaf3d4e7f5b4c994 [term 1 LEADER]: Becoming Leader. State: Replica: 8a2b9515eba149bcbaf3d4e7f5b4c994, State: Running, Role: LEADER
I20260620 18:37:03.286609 30146 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a2b9515eba149bcbaf3d4e7f5b4c994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a2b9515eba149bcbaf3d4e7f5b4c994" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 36063 } }
I20260620 18:37:03.287259 30141 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a2b9515eba149bcbaf3d4e7f5b4c994 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:37:03.289501 30148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a2b9515eba149bcbaf3d4e7f5b4c994 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a2b9515eba149bcbaf3d4e7f5b4c994. Latest consensus state: current_term: 1 leader_uuid: "8a2b9515eba149bcbaf3d4e7f5b4c994" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a2b9515eba149bcbaf3d4e7f5b4c994" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 36063 } } }
I20260620 18:37:03.289881 30148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a2b9515eba149bcbaf3d4e7f5b4c994 [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:03.290095 30147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a2b9515eba149bcbaf3d4e7f5b4c994 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a2b9515eba149bcbaf3d4e7f5b4c994" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a2b9515eba149bcbaf3d4e7f5b4c994" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 36063 } } }
I20260620 18:37:03.290383 30147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a2b9515eba149bcbaf3d4e7f5b4c994 [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:03.291416 30155 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:37:03.296571 30155 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:37:03.305994 30155 catalog_manager.cc:1347] Generated new cluster ID: 4086d56566f84512aa7c74f3e8e82756
I20260620 18:37:03.306121 30155 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:37:03.330360 30155 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:37:03.332119 30155 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:37:03.341499 30155 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a2b9515eba149bcbaf3d4e7f5b4c994: Generated new TSK 0
I20260620 18:37:03.342346 30155 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.131.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab.
Entry for principal kudu/127.10.131.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab.
I20260620 18:37:03.402319 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.1:0
--local_ip_for_outbound_sockets=127.10.131.1
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:36063
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:42211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:03.737180 30167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:03.737473 30167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:03.737682 30167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:03.745162 30167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:03.745298 30167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:03.745505 30167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.1
I20260620 18:37:03.755600 30167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:42211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.1.keytab
--principal=kudu/127.10.131.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:36063
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.1
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:37:03.756989 30167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:03.758342 30167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:03.771935 30172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:03.772020 30175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:03.771984 30173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:03.773260 30167 server_base.cc:1034] running on GCE node
I20260620 18:37:03.773947 30167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:03.774991 30167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:03.776221 30167 hybrid_clock.cc:648] HybridClock initialized: now 1781980623776152 us; error 86 us; skew 500 ppm
I20260620 18:37:03.776533 30167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:03 dist-test-slave-cbn3 krb5kdc[30061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980623, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:03.779675 30167 init.cc:377] Logged in from keytab as kudu/127.10.131.1@KRBTEST.COM (short username kudu)
I20260620 18:37:03.781147 30167 webserver.cc:460] Webserver started at http://127.10.131.1:37603/ using document root <none> and password file <none>
I20260620 18:37:03.781814 30167 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:03.781946 30167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:03.782262 30167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:03.785169 30167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-0/data/instance:
uuid: "4992d2f13b0744c690f509ad55cff4e9"
format_stamp: "Formatted at 2026-06-20 18:37:03 on dist-test-slave-cbn3"
server_key: "1c2a72012cffaadc82c613b2f6499607"
server_key_iv: "7d8a2711aa101269fbc0f9d7e83134cd"
server_key_version: "encryptionkey@0"
I20260620 18:37:03.785926 30167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance:
uuid: "4992d2f13b0744c690f509ad55cff4e9"
format_stamp: "Formatted at 2026-06-20 18:37:03 on dist-test-slave-cbn3"
server_key: "1c2a72012cffaadc82c613b2f6499607"
server_key_iv: "7d8a2711aa101269fbc0f9d7e83134cd"
server_key_version: "encryptionkey@0"
I20260620 18:37:03.791239 30167 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:37:03.794615 30182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:03.795887 30167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:37:03.796053 30167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-0/wal
uuid: "4992d2f13b0744c690f509ad55cff4e9"
format_stamp: "Formatted at 2026-06-20 18:37:03 on dist-test-slave-cbn3"
server_key: "1c2a72012cffaadc82c613b2f6499607"
server_key_iv: "7d8a2711aa101269fbc0f9d7e83134cd"
server_key_version: "encryptionkey@0"
I20260620 18:37:03.796238 30167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:03.810842 30167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:03.814904 30167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:03.815186 30167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:03.816102 30167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:37:03.817479 30167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:37:03.817550 30167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:03.817720 30167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:37:03.817780 30167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:03.850114 30167 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.1:41599
I20260620 18:37:03.850159 30294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.1:41599 every 8 connection(s)
I20260620 18:37:03.851433 30167 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
I20260620 18:37:03.860880 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 30167
I20260620 18:37:03.861073 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance
I20260620 18:37:03.861533 10764 external_mini_cluster.cc:1428] Setting key 3600582b06d580f6a8ec3998dc63bc2d
Jun 20 18:37:03 dist-test-slave-cbn3 krb5kdc[30061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980623, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.1@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:37:03.872491 30297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:03.853914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:33157 (local address 127.10.131.62:36063)
0620 18:37:03.854325 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:03.854330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:03.855253 (+ 923us) server_negotiation.cc:407] Connection header received
0620 18:37:03.857097 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:03.857102 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:03.857196 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:03.857310 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:03.858739 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:03.859454 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:03.861272 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:03.861746 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:03.861859 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:03.866069 (+ 4210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:03.866075 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:03.866079 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:03.866115 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:03.868807 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:03.869832 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:03.869837 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:03.869839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:03.869964 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:03.870479 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:03.870482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:03.870485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:03.870748 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:03.870872 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:03.871771 (+ 899us) server_negotiation.cc:299] Negotiation successful
0620 18:37:03.872059 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":205,"threads_started":1}
I20260620 18:37:03.873260 30298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:03.854427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:36063 (local address 127.10.131.1:33157)
0620 18:37:03.855096 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:03.855135 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:03.856780 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:03.857542 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:03.857550 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:03.858045 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:03.858575 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:03.858592 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:03.859620 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:03.859624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:03.861136 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:03.861152 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:03.862016 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:03.862023 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:03.862259 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:03.862920 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:03.862947 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:03.865830 (+ 2883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:03.869039 (+ 3209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:03.869052 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:03.869074 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:03.869697 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:03.870073 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:03.870080 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:03.870086 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:03.870337 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:03.870881 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:03.870893 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:03.871366 (+ 473us) client_negotiation.cc:769] Sending connection context
0620 18:37:03.871664 (+ 298us) client_negotiation.cc:240] Negotiation successful
0620 18:37:03.872148 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":263,"threads_started":1}
I20260620 18:37:03.874794 30295 heartbeater.cc:347] Connected to a master server at 127.10.131.62:36063
I20260620 18:37:03.875121 30295 heartbeater.cc:464] Registering TS with master...
I20260620 18:37:03.875829 30295 heartbeater.cc:511] Master 127.10.131.62:36063 requested a full tablet report, sending...
I20260620 18:37:03.877801 30106 ts_manager.cc:194] Registered new tserver with Master: 4992d2f13b0744c690f509ad55cff4e9 (127.10.131.1:41599)
I20260620 18:37:03.879210 30106 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.1@KRBTEST.COM'} at 127.10.131.1:33157
WARNING: no policy specified for kudu/127.10.131.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab.
Entry for principal kudu/127.10.131.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab.
I20260620 18:37:03.915482 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.2:0
--local_ip_for_outbound_sockets=127.10.131.2
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:36063
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:42211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:04.231300 30301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:04.231560 30301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:04.231727 30301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:04.238790 30301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:04.238914 30301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:04.239095 30301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.2
I20260620 18:37:04.248607 30301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:42211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.2.keytab
--principal=kudu/127.10.131.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:36063
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.2
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:37:04.249814 30301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:04.251003 30301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:04.264092 30309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:04.264130 30307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:04.264174 30306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:04.270160 30301 server_base.cc:1034] running on GCE node
I20260620 18:37:04.270756 30301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:04.271668 30301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:04.272943 30301 hybrid_clock.cc:648] HybridClock initialized: now 1781980624272877 us; error 52 us; skew 500 ppm
I20260620 18:37:04.273243 30301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:04 dist-test-slave-cbn3 krb5kdc[30061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980624, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:04.279740 30301 init.cc:377] Logged in from keytab as kudu/127.10.131.2@KRBTEST.COM (short username kudu)
I20260620 18:37:04.281253 30301 webserver.cc:460] Webserver started at http://127.10.131.2:42385/ using document root <none> and password file <none>
I20260620 18:37:04.281934 30301 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:04.282078 30301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:04.282407 30301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:04.285264 30301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-1/data/instance:
uuid: "0f0e44a8ffcf4ea8bcd3d74084d3af46"
format_stamp: "Formatted at 2026-06-20 18:37:04 on dist-test-slave-cbn3"
server_key: "f057e2a9fb2f6e46051479151115f1d3"
server_key_iv: "233210b3f1a11562fdc340e17da09c33"
server_key_version: "encryptionkey@0"
I20260620 18:37:04.286002 30301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance:
uuid: "0f0e44a8ffcf4ea8bcd3d74084d3af46"
format_stamp: "Formatted at 2026-06-20 18:37:04 on dist-test-slave-cbn3"
server_key: "f057e2a9fb2f6e46051479151115f1d3"
server_key_iv: "233210b3f1a11562fdc340e17da09c33"
server_key_version: "encryptionkey@0"
I20260620 18:37:04.291196 30301 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:37:04.294773 30316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:04.296033 30301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 18:37:04.296196 30301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-1/wal
uuid: "0f0e44a8ffcf4ea8bcd3d74084d3af46"
format_stamp: "Formatted at 2026-06-20 18:37:04 on dist-test-slave-cbn3"
server_key: "f057e2a9fb2f6e46051479151115f1d3"
server_key_iv: "233210b3f1a11562fdc340e17da09c33"
server_key_version: "encryptionkey@0"
I20260620 18:37:04.296382 30301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:04.341490 30301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:04.345204 30301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:04.345468 30301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:04.346362 30301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:37:04.347612 30301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:37:04.347682 30301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:04.347855 30301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:37:04.347904 30301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:04.381260 30301 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.2:39441
I20260620 18:37:04.381305 30428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.2:39441 every 8 connection(s)
I20260620 18:37:04.382575 30301 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
I20260620 18:37:04.389243 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 30301
I20260620 18:37:04.389441 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance
I20260620 18:37:04.389858 10764 external_mini_cluster.cc:1428] Setting key da7dc883d105446c2f3e533f3b3fdbf9
Jun 20 18:37:04 dist-test-slave-cbn3 krb5kdc[30061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980624, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.2@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:37:04.402992 30431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:04.385057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:59711 (local address 127.10.131.62:36063)
0620 18:37:04.385494 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:04.385498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:04.386246 (+ 748us) server_negotiation.cc:407] Connection header received
0620 18:37:04.388129 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:04.388134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:04.388230 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:04.388360 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:04.389962 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.390905 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.392643 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.393123 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.393251 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:04.396674 (+ 3423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:04.396680 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:04.396692 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:04.396735 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:04.399719 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:04.400541 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:04.400546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:04.400550 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:04.400727 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:04.401226 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:04.401231 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:04.401234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:04.401577 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:04.401735 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:04.402421 (+ 686us) server_negotiation.cc:299] Negotiation successful
0620 18:37:04.402633 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":229,"threads_started":1}
I20260620 18:37:04.403959 30432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:04.385470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:36063 (local address 127.10.131.2:59711)
0620 18:37:04.386097 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:04.386131 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:04.387849 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:04.388619 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:04.388631 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:04.389180 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:04.389799 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:04.389817 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.391095 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.391099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:04.392501 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:04.392513 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.393326 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.393333 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:04.393555 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:04.394211 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:04.394232 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:04.396462 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:04.399848 (+ 3386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:04.399858 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:04.399875 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:04.400423 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:04.400826 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:04.400831 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:04.400834 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:04.401084 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:04.401709 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:04.401716 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:04.402053 (+ 337us) client_negotiation.cc:769] Sending connection context
0620 18:37:04.402311 (+ 258us) client_negotiation.cc:240] Negotiation successful
0620 18:37:04.402824 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":256,"threads_started":1}
I20260620 18:37:04.405371 30429 heartbeater.cc:347] Connected to a master server at 127.10.131.62:36063
I20260620 18:37:04.405689 30429 heartbeater.cc:464] Registering TS with master...
I20260620 18:37:04.406527 30429 heartbeater.cc:511] Master 127.10.131.62:36063 requested a full tablet report, sending...
I20260620 18:37:04.408078 30106 ts_manager.cc:194] Registered new tserver with Master: 0f0e44a8ffcf4ea8bcd3d74084d3af46 (127.10.131.2:39441)
I20260620 18:37:04.409396 30106 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.2@KRBTEST.COM'} at 127.10.131.2:59711
WARNING: no policy specified for kudu/127.10.131.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab.
Entry for principal kudu/127.10.131.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab.
I20260620 18:37:04.442158 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.3:0
--local_ip_for_outbound_sockets=127.10.131.3
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:36063
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:42211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:04.765511 30435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:04.765832 30435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:04.766013 30435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:04.773552 30435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:04.773706 30435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:04.773886 30435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.3
I20260620 18:37:04.784246 30435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:42211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.3.keytab
--principal=kudu/127.10.131.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:36063
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.3
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:37:04.785631 30435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:04.786967 30435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:04.800607 30441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:04.800559 30443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:04.801252 30440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:04.801573 30435 server_base.cc:1034] running on GCE node
I20260620 18:37:04.802192 30435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:04.803138 30435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:04.804368 30435 hybrid_clock.cc:648] HybridClock initialized: now 1781980624804327 us; error 71 us; skew 500 ppm
I20260620 18:37:04.804679 30435 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:04 dist-test-slave-cbn3 krb5kdc[30061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980624, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:04.807713 30435 init.cc:377] Logged in from keytab as kudu/127.10.131.3@KRBTEST.COM (short username kudu)
I20260620 18:37:04.809244 30435 webserver.cc:460] Webserver started at http://127.10.131.3:42867/ using document root <none> and password file <none>
I20260620 18:37:04.809906 30435 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:04.810047 30435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:04.810357 30435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:04.813257 30435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-2/data/instance:
uuid: "64769abce709430683bbb28596200d30"
format_stamp: "Formatted at 2026-06-20 18:37:04 on dist-test-slave-cbn3"
server_key: "6792618105e2bda2fe6e6f5d9a710fbc"
server_key_iv: "ce965c78a9e4dceffddfc3b1fe72fde2"
server_key_version: "encryptionkey@0"
I20260620 18:37:04.814010 30435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance:
uuid: "64769abce709430683bbb28596200d30"
format_stamp: "Formatted at 2026-06-20 18:37:04 on dist-test-slave-cbn3"
server_key: "6792618105e2bda2fe6e6f5d9a710fbc"
server_key_iv: "ce965c78a9e4dceffddfc3b1fe72fde2"
server_key_version: "encryptionkey@0"
I20260620 18:37:04.819023 30435 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:37:04.822429 30450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:04.823664 30435 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:37:04.823835 30435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-2/wal
uuid: "64769abce709430683bbb28596200d30"
format_stamp: "Formatted at 2026-06-20 18:37:04 on dist-test-slave-cbn3"
server_key: "6792618105e2bda2fe6e6f5d9a710fbc"
server_key_iv: "ce965c78a9e4dceffddfc3b1fe72fde2"
server_key_version: "encryptionkey@0"
I20260620 18:37:04.824033 30435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:04.843509 30435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:04.847380 30435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:04.847668 30435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:04.848596 30435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:37:04.850078 30435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:37:04.850188 30435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:04.850339 30435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:37:04.850411 30435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:04.882057 30295 heartbeater.cc:503] Master 127.10.131.62:36063 was elected leader, sending a full tablet report...
I20260620 18:37:04.883462 30435 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.3:41103
I20260620 18:37:04.883574 30562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.3:41103 every 8 connection(s)
I20260620 18:37:04.884951 30435 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
I20260620 18:37:04.895182 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 30435
I20260620 18:37:04.895416 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance
I20260620 18:37:04.895915 10764 external_mini_cluster.cc:1428] Setting key 4db84bab2fc89788d4444577b05b2596
Jun 20 18:37:04 dist-test-slave-cbn3 krb5kdc[30061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980624, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.3@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:37:04.904424 30431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:04.887363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:42071 (local address 127.10.131.62:36063)
0620 18:37:04.887559 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:04.887566 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:04.888707 (+ 1141us) server_negotiation.cc:407] Connection header received
0620 18:37:04.890649 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:04.890655 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:04.890742 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:04.890829 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:04.892443 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.893228 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.894729 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.895093 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.895214 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:04.899058 (+ 3844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:04.899062 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:04.899066 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:04.899097 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:04.901985 (+ 2888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:04.902619 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:04.902623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:04.902625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:04.902738 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:04.903091 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:04.903094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:04.903096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:04.903346 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:04.903466 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:04.904039 (+ 573us) server_negotiation.cc:299] Negotiation successful
0620 18:37:04.904197 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:37:04.905488 30565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:04.887853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:36063 (local address 127.10.131.3:42071)
0620 18:37:04.888561 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:04.888597 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:04.890246 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:04.891084 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:04.891092 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:04.891648 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:04.892278 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:04.892297 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.893371 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.893375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:04.894608 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:04.894621 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.895338 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.895346 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:04.895609 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:04.896296 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:04.896317 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:04.898884 (+ 2567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:04.902110 (+ 3226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:04.902116 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:04.902127 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:04.902525 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:04.902835 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:04.902838 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:04.902840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:04.903004 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:04.903456 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:04.903462 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:04.903739 (+ 277us) client_negotiation.cc:769] Sending connection context
0620 18:37:04.903954 (+ 215us) client_negotiation.cc:240] Negotiation successful
0620 18:37:04.904306 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":285,"threads_started":1}
I20260620 18:37:04.906792 30563 heartbeater.cc:347] Connected to a master server at 127.10.131.62:36063
I20260620 18:37:04.907091 30563 heartbeater.cc:464] Registering TS with master...
I20260620 18:37:04.907744 30563 heartbeater.cc:511] Master 127.10.131.62:36063 requested a full tablet report, sending...
I20260620 18:37:04.909162 30106 ts_manager.cc:194] Registered new tserver with Master: 64769abce709430683bbb28596200d30 (127.10.131.3:41103)
I20260620 18:37:04.910252 30106 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.131.3@KRBTEST.COM'} at 127.10.131.3:42071
I20260620 18:37:04.911001 10764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:37:04.926841 30431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:04.914815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42336 (local address 127.10.131.62:36063)
0620 18:37:04.914976 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:04.914982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:04.915407 (+ 425us) server_negotiation.cc:407] Connection header received
0620 18:37:04.915596 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:04.915601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:04.915671 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:04.915777 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:04.917111 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.917821 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.919085 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.919510 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.919639 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:04.921934 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:04.921940 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:04.921945 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:04.921983 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:04.924189 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:04.924866 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:04.924873 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:04.924876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:04.925000 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:04.925370 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:04.925375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:04.925379 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:04.925685 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:04.925965 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:04.926276 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 18:37:04.926483 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 18:37:04.930330 30106 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:37:04.937119 30106 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:37:04.959888 30577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:04.949405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:39441 (local address 127.0.0.1:43974)
0620 18:37:04.950705 (+ 1300us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:04.950725 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:04.950843 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:04.952286 (+ 1443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:04.952296 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:04.952330 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:37:04.952891 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:04.952904 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.954735 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.954745 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:04.957571 (+ 2826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:04.957581 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.958827 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.958833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:04.959016 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:04.959071 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 18:37:04.959222 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 18:37:04.959444 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1140,"mutex_wait_us":182,"thread_start_us":966,"threads_started":1}
I20260620 18:37:04.960040 30578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:04.949960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43974 (local address 127.10.131.2:39441)
0620 18:37:04.951403 (+ 1443us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:04.951416 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:04.951444 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:37:04.951538 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:04.951551 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:04.951817 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:04.952007 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:37:04.953079 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.954554 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.957734 (+ 3180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.958633 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.958809 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:04.959002 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:04.959541 (+ 539us) server_negotiation.cc:299] Negotiation successful
0620 18:37:04.959716 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1232,"thread_start_us":1112,"threads_started":1}
I20260620 18:37:04.960480 30576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:04.949000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33762 (local address 127.10.131.3:41103)
0620 18:37:04.951763 (+ 2763us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:04.951772 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:04.951804 (+ 32us) server_negotiation.cc:407] Connection header received
0620 18:37:04.951895 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:04.951902 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:04.952113 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:04.953050 (+ 937us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:37:04.953361 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.954475 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.957707 (+ 3232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.959076 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.959203 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:04.959430 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:04.959995 (+ 565us) server_negotiation.cc:299] Negotiation successful
0620 18:37:04.960157 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2560,"thread_start_us":2394,"threads_started":1}
I20260620 18:37:04.961005 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:04.948664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:41103 (local address 127.0.0.1:33762)
0620 18:37:04.949772 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:04.949828 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:04.950042 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:04.952314 (+ 2272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:04.952319 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:04.952346 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:37:04.952796 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:04.952809 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.954600 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.954608 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:04.957537 (+ 2929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:04.957553 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.959275 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.959282 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:04.959491 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:04.959520 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 18:37:04.959610 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 18:37:04.959728 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"mutex_wait_us":218,"thread_start_us":1131,"threads_started":1}
I20260620 18:37:04.961100 30579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:04.950703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:41599 (local address 127.0.0.1:45428)
0620 18:37:04.951998 (+ 1295us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:04.952015 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:04.952112 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:04.953225 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:04.953232 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:04.953266 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:37:04.953587 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:04.953597 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.955233 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.955239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:04.957325 (+ 2086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:04.957337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.959429 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.959435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:04.960409 (+ 974us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:04.960454 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 18:37:04.960537 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 18:37:04.960661 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1175,"mutex_wait_us":3,"thread_start_us":1010,"threads_started":1}
I20260620 18:37:04.961287 30580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:04.951354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45428 (local address 127.10.131.1:41599)
0620 18:37:04.952611 (+ 1257us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:04.952621 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:04.952649 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:37:04.952765 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:04.952774 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:04.953035 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:04.953223 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:37:04.953774 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.955104 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.957471 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:04.958909 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:04.959045 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:04.959274 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:04.960761 (+ 1487us) server_negotiation.cc:299] Negotiation successful
0620 18:37:04.960955 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1085,"thread_start_us":1539,"threads_started":1}
I20260620 18:37:04.963963 30364 tablet_service.cc:1467] Processing CreateTablet for tablet f6aebc4627a54ef594ab500b5ee7e9b2 (DEFAULT_TABLE table=test-table [id=0f967ad37cc7461082625fbef752491b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:37:04.965323 30364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6aebc4627a54ef594ab500b5ee7e9b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:04.964797 30498 tablet_service.cc:1467] Processing CreateTablet for tablet f6aebc4627a54ef594ab500b5ee7e9b2 (DEFAULT_TABLE table=test-table [id=0f967ad37cc7461082625fbef752491b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:37:04.964797 30230 tablet_service.cc:1467] Processing CreateTablet for tablet f6aebc4627a54ef594ab500b5ee7e9b2 (DEFAULT_TABLE table=test-table [id=0f967ad37cc7461082625fbef752491b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:37:04.966197 30230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6aebc4627a54ef594ab500b5ee7e9b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:04.966208 30498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6aebc4627a54ef594ab500b5ee7e9b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:04.976625 30581 tablet_bootstrap.cc:492] T f6aebc4627a54ef594ab500b5ee7e9b2 P 0f0e44a8ffcf4ea8bcd3d74084d3af46: Bootstrap starting.
I20260620 18:37:04.980400 30581 tablet_bootstrap.cc:654] T f6aebc4627a54ef594ab500b5ee7e9b2 P 0f0e44a8ffcf4ea8bcd3d74084d3af46: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:04.980587 30583 tablet_bootstrap.cc:492] T f6aebc4627a54ef594ab500b5ee7e9b2 P 4992d2f13b0744c690f509ad55cff4e9: Bootstrap starting.
I20260620 18:37:04.981642 30582 tablet_bootstrap.cc:492] T f6aebc4627a54ef594ab500b5ee7e9b2 P 64769abce709430683bbb28596200d30: Bootstrap starting.
I20260620 18:37:04.982538 30581 log.cc:824] T f6aebc4627a54ef594ab500b5ee7e9b2 P 0f0e44a8ffcf4ea8bcd3d74084d3af46: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:04.984350 30583 tablet_bootstrap.cc:654] T f6aebc4627a54ef594ab500b5ee7e9b2 P 4992d2f13b0744c690f509ad55cff4e9: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:04.984884 30582 tablet_bootstrap.cc:654] T f6aebc4627a54ef594ab500b5ee7e9b2 P 64769abce709430683bbb28596200d30: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:04.985622 30583 log.cc:824] T f6aebc4627a54ef594ab500b5ee7e9b2 P 4992d2f13b0744c690f509ad55cff4e9: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:04.985823 30581 tablet_bootstrap.cc:492] T f6aebc4627a54ef594ab500b5ee7e9b2 P 0f0e44a8ffcf4ea8bcd3d74084d3af46: No bootstrap required, opened a new log
I20260620 18:37:04.985859 30582 log.cc:824] T f6aebc4627a54ef594ab500b5ee7e9b2 P 64769abce709430683bbb28596200d30: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:04.986088 30581 ts_tablet_manager.cc:1397] T f6aebc4627a54ef594ab500b5ee7e9b2 P 0f0e44a8ffcf4ea8bcd3d74084d3af46: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 18:37:04.988585 30583 tablet_bootstrap.cc:492] T f6aebc4627a54ef594ab500b5ee7e9b2 P 4992d2f13b0744c690f509ad55cff4e9: No bootstrap required, opened a new log
I20260620 18:37:04.988873 30583 ts_tablet_manager.cc:1397] T f6aebc4627a54ef594ab500b5ee7e9b2 P 4992d2f13b0744c690f509ad55cff4e9: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20260620 18:37:04.989982 30582 tablet_bootstrap.cc:492] T f6aebc4627a54ef594ab500b5ee7e9b2 P 64769abce709430683bbb28596200d30: No bootstrap required, opened a new log
I20260620 18:37:04.990159 30582 ts_tablet_manager.cc:1397] T f6aebc4627a54ef594ab500b5ee7e9b2 P 64769abce709430683bbb28596200d30: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260620 18:37:04.997196 30581 raft_consensus.cc:348] T f6aebc4627a54ef594ab500b5ee7e9b2 P 0f0e44a8ffcf4ea8bcd3d74084d3af46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64769abce709430683bbb28596200d30" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 41103 } } peers { permanent_uuid: "4992d2f13b0744c690f509ad55cff4e9" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41599 } } peers { permanent_uuid: "0f0e44a8ffcf4ea8bcd3d74084d3af46" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39441 } }
I20260620 18:37:04.997695 30581 raft_consensus.cc:374] T f6aebc4627a54ef594ab500b5ee7e9b2 P 0f0e44a8ffcf4ea8bcd3d74084d3af46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:04.997830 30581 raft_consensus.cc:729] T f6aebc4627a54ef594ab500b5ee7e9b2 P 0f0e44a8ffcf4ea8bcd3d74084d3af46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f0e44a8ffcf4ea8bcd3d74084d3af46, State: Initialized, Role: FOLLOWER
I20260620 18:37:04.997687 30582 raft_consensus.cc:348] T f6aebc4627a54ef594ab500b5ee7e9b2 P 64769abce709430683bbb28596200d30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64769abce709430683bbb28596200d30" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 41103 } } peers { permanent_uuid: "4992d2f13b0744c690f509ad55cff4e9" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41599 } } peers { permanent_uuid: "0f0e44a8ffcf4ea8bcd3d74084d3af46" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39441 } }
I20260620 18:37:04.998096 30582 raft_consensus.cc:374] T f6aebc4627a54ef594ab500b5ee7e9b2 P 64769abce709430683bbb28596200d30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:04.998246 30582 raft_consensus.cc:729] T f6aebc4627a54ef594ab500b5ee7e9b2 P 64769abce709430683bbb28596200d30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64769abce709430683bbb28596200d30, State: Initialized, Role: FOLLOWER
I20260620 18:37:04.998708 30581 consensus_queue.cc:260] T f6aebc4627a54ef594ab500b5ee7e9b2 P 0f0e44a8ffcf4ea8bcd3d74084d3af46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64769abce709430683bbb28596200d30" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 41103 } } peers { permanent_uuid: "4992d2f13b0744c690f509ad55cff4e9" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41599 } } peers { permanent_uuid: "0f0e44a8ffcf4ea8bcd3d74084d3af46" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39441 } }
I20260620 18:37:04.998999 30582 consensus_queue.cc:260] T f6aebc4627a54ef594ab500b5ee7e9b2 P 64769abce709430683bbb28596200d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64769abce709430683bbb28596200d30" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 41103 } } peers { permanent_uuid: "4992d2f13b0744c690f509ad55cff4e9" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41599 } } peers { permanent_uuid: "0f0e44a8ffcf4ea8bcd3d74084d3af46" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39441 } }
I20260620 18:37:04.999514 30583 raft_consensus.cc:348] T f6aebc4627a54ef594ab500b5ee7e9b2 P 4992d2f13b0744c690f509ad55cff4e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64769abce709430683bbb28596200d30" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 41103 } } peers { permanent_uuid: "4992d2f13b0744c690f509ad55cff4e9" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41599 } } peers { permanent_uuid: "0f0e44a8ffcf4ea8bcd3d74084d3af46" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39441 } }
I20260620 18:37:04.999976 30583 raft_consensus.cc:374] T f6aebc4627a54ef594ab500b5ee7e9b2 P 4992d2f13b0744c690f509ad55cff4e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:05.000097 30583 raft_consensus.cc:729] T f6aebc4627a54ef594ab500b5ee7e9b2 P 4992d2f13b0744c690f509ad55cff4e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4992d2f13b0744c690f509ad55cff4e9, State: Initialized, Role: FOLLOWER
I20260620 18:37:05.000905 30583 consensus_queue.cc:260] T f6aebc4627a54ef594ab500b5ee7e9b2 P 4992d2f13b0744c690f509ad55cff4e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64769abce709430683bbb28596200d30" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 41103 } } peers { permanent_uuid: "4992d2f13b0744c690f509ad55cff4e9" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41599 } } peers { permanent_uuid: "0f0e44a8ffcf4ea8bcd3d74084d3af46" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39441 } }
I20260620 18:37:05.001780 30563 heartbeater.cc:503] Master 127.10.131.62:36063 was elected leader, sending a full tablet report...
I20260620 18:37:05.002171 30582 ts_tablet_manager.cc:1428] T f6aebc4627a54ef594ab500b5ee7e9b2 P 64769abce709430683bbb28596200d30: Time spent starting tablet: real 0.012s user 0.006s sys 0.006s
I20260620 18:37:05.004336 30429 heartbeater.cc:503] Master 127.10.131.62:36063 was elected leader, sending a full tablet report...
I20260620 18:37:05.004550 30583 ts_tablet_manager.cc:1428] T f6aebc4627a54ef594ab500b5ee7e9b2 P 4992d2f13b0744c690f509ad55cff4e9: Time spent starting tablet: real 0.016s user 0.007s sys 0.006s
I20260620 18:37:05.005901 30581 ts_tablet_manager.cc:1428] T f6aebc4627a54ef594ab500b5ee7e9b2 P 0f0e44a8ffcf4ea8bcd3d74084d3af46: Time spent starting tablet: real 0.020s user 0.016s sys 0.000s
W20260620 18:37:05.104031 30296 tablet.cc:2369] T f6aebc4627a54ef594ab500b5ee7e9b2 P 4992d2f13b0744c690f509ad55cff4e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:37:05.134792 30430 tablet.cc:2369] T f6aebc4627a54ef594ab500b5ee7e9b2 P 0f0e44a8ffcf4ea8bcd3d74084d3af46: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:37:05.136029 30587 raft_consensus.cc:482] T f6aebc4627a54ef594ab500b5ee7e9b2 P 0f0e44a8ffcf4ea8bcd3d74084d3af46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:37:05.136312 30587 raft_consensus.cc:504] T f6aebc4627a54ef594ab500b5ee7e9b2 P 0f0e44a8ffcf4ea8bcd3d74084d3af46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64769abce709430683bbb28596200d30" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 41103 } } peers { permanent_uuid: "4992d2f13b0744c690f509ad55cff4e9" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41599 } } peers { permanent_uuid: "0f0e44a8ffcf4ea8bcd3d74084d3af46" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39441 } }
W20260620 18:37:05.136463 30564 tablet.cc:2369] T f6aebc4627a54ef594ab500b5ee7e9b2 P 64769abce709430683bbb28596200d30: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:37:05.137972 30587 leader_election.cc:288] T f6aebc4627a54ef594ab500b5ee7e9b2 P 0f0e44a8ffcf4ea8bcd3d74084d3af46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64769abce709430683bbb28596200d30 (127.10.131.3:41103), 4992d2f13b0744c690f509ad55cff4e9 (127.10.131.1:41599)
I20260620 18:37:05.142695 30588 raft_consensus.cc:482] T f6aebc4627a54ef594ab500b5ee7e9b2 P 64769abce709430683bbb28596200d30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:37:05.143011 30588 raft_consensus.cc:504] T f6aebc4627a54ef594ab500b5ee7e9b2 P 64769abce709430683bbb28596200d30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64769abce709430683bbb28596200d30" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 41103 } } peers { permanent_uuid: "4992d2f13b0744c690f509ad55cff4e9" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41599 } } peers { permanent_uuid: "0f0e44a8ffcf4ea8bcd3d74084d3af46" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39441 } }
I20260620 18:37:05.145458 30576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:05.138473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:52405 (local address 127.10.131.3:41103)
0620 18:37:05.138721 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:05.138727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:05.139045 (+ 318us) server_negotiation.cc:407] Connection header received
0620 18:37:05.139334 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:05.139341 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:05.139424 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:05.139487 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:37:05.140233 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:05.141286 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:05.143576 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:05.144435 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:05.144615 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:05.144709 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:05.145062 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 18:37:05.145198 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 18:37:05.146211 30590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:05.138292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:41103 (local address 127.10.131.2:52405)
0620 18:37:05.138910 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:05.138949 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:05.139162 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:05.139650 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:05.139653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:05.139673 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:37:05.140058 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:05.140071 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:05.141403 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:05.141409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:05.143412 (+ 2003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:05.143430 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:05.144670 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:05.144676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:05.144830 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:05.144911 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 18:37:05.145606 (+ 695us) client_negotiation.cc:240] Negotiation successful
0620 18:37:05.145786 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"mutex_wait_us":1,"thread_start_us":335,"threads_started":1}
I20260620 18:37:05.146270 30580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:05.139174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:38299 (local address 127.10.131.1:41599)
0620 18:37:05.139379 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:05.139384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:05.139942 (+ 558us) server_negotiation.cc:407] Connection header received
0620 18:37:05.140050 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:05.140055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:05.140138 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:05.140226 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:37:05.140828 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:05.141782 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:05.143797 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:05.144972 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:05.145137 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:05.145216 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:05.145805 (+ 589us) server_negotiation.cc:299] Negotiation successful
0620 18:37:05.145949 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 18:37:05.147307 30518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6aebc4627a54ef594ab500b5ee7e9b2" candidate_uuid: "0f0e44a8ffcf4ea8bcd3d74084d3af46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64769abce709430683bbb28596200d30" is_pre_election: true
I20260620 18:37:05.147784 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:05.139012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:41599 (local address 127.10.131.2:38299)
0620 18:37:05.139821 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:05.139838 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:05.139937 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:05.140268 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:05.140272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:05.140299 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:37:05.140676 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:05.140686 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:05.141917 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:05.141921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:05.143634 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:05.143647 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:05.145350 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:05.145356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:05.145568 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:05.145613 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 18:37:05.147068 (+ 1455us) client_negotiation.cc:240] Negotiation successful
0620 18:37:05.147256 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"mutex_wait_us":3,"thread_start_us":524,"threads_started":1}
I20260620 18:37:05.147907 30518 raft_consensus.cc:2454] T f6aebc4627a54ef594ab500b5ee7e9b2 P 64769abce709430683bbb28596200d30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f0e44a8ffcf4ea8bcd3d74084d3af46 in term 0.
I20260620 18:37:05.148061 30588 leader_election.cc:288] T f6aebc4627a54ef594ab500b5ee7e9b2 P 64769abce709430683bbb28596200d30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4992d2f13b0744c690f509ad55cff4e9 (127.10.131.1:41599), 0f0e44a8ffcf4ea8bcd3d74084d3af46 (127.10.131.2:39441)
I20260620 18:37:05.148862 30317 leader_election.cc:302] T f6aebc4627a54ef594ab500b5ee7e9b2 P 0f0e44a8ffcf4ea8bcd3d74084d3af46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f0e44a8ffcf4ea8bcd3d74084d3af46, 64769abce709430683bbb28596200d30; no voters:
I20260620 18:37:05.149271 30587 raft_consensus.cc:2781] T f6aebc4627a54ef594ab500b5ee7e9b2 P 0f0e44a8ffcf4ea8bcd3d74084d3af46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:37:05.149411 30587 raft_consensus.cc:482] T f6aebc4627a54ef594ab500b5ee7e9b2 P 0f0e44a8ffcf4ea8bcd3d74084d3af46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:37:05.149524 30587 raft_consensus.cc:3037] T f6aebc4627a54ef594ab500b5ee7e9b2 P 0f0e44a8ffcf4ea8bcd3d74084d3af46 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:05.149942 30250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6aebc4627a54ef594ab500b5ee7e9b2" candidate_uuid: "0f0e44a8ffcf4ea8bcd3d74084d3af46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4992d2f13b0744c690f509ad55cff4e9" is_pre_election: true
I20260620 18:37:05.150434 30250 raft_consensus.cc:2454] T f6aebc4627a54ef594ab500b5ee7e9b2 P 4992d2f13b0744c690f509ad55cff4e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f0e44a8ffcf4ea8bcd3d74084d3af46 in term 0.
I20260620 18:37:05.152922 30587 raft_consensus.cc:504] T f6aebc4627a54ef594ab500b5ee7e9b2 P 0f0e44a8ffcf4ea8bcd3d74084d3af46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64769abce709430683bbb28596200d30" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 41103 } } peers { permanent_uuid: "4992d2f13b0744c690f509ad55cff4e9" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41599 } } peers { permanent_uuid: "0f0e44a8ffcf4ea8bcd3d74084d3af46" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39441 } }
I20260620 18:37:05.153497 30565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:05.145140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:41599 (local address 127.10.131.3:59579)
0620 18:37:05.145460 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:05.145478 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:05.145575 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:05.146939 (+ 1364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:05.146943 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:05.146978 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:37:05.147165 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:05.147179 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:05.149731 (+ 2552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:05.149737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:05.151570 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:05.151581 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:05.152662 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:05.152669 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:05.152933 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:05.152956 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 18:37:05.153062 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 18:37:05.153189 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":1}
I20260620 18:37:05.153672 30587 leader_election.cc:288] T f6aebc4627a54ef594ab500b5ee7e9b2 P 0f0e44a8ffcf4ea8bcd3d74084d3af46 [CANDIDATE]: Term 1 election: Requested vote from peers 64769abce709430683bbb28596200d30 (127.10.131.3:41103), 4992d2f13b0744c690f509ad55cff4e9 (127.10.131.1:41599)
I20260620 18:37:05.153992 30518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6aebc4627a54ef594ab500b5ee7e9b2" candidate_uuid: "0f0e44a8ffcf4ea8bcd3d74084d3af46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64769abce709430683bbb28596200d30"
I20260620 18:37:05.154179 30518 raft_consensus.cc:3037] T f6aebc4627a54ef594ab500b5ee7e9b2 P 64769abce709430683bbb28596200d30 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:05.154429 30580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:05.145870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:59579 (local address 127.10.131.1:41599)
0620 18:37:05.146631 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:05.146637 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:05.146657 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:37:05.146721 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:05.146728 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:05.146827 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:05.146928 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:37:05.147965 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:05.149561 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:05.151720 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:05.152508 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:05.152676 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:05.152742 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:05.153083 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 18:37:05.153238 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":2929,"threads_started":1}
I20260620 18:37:05.155014 30250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6aebc4627a54ef594ab500b5ee7e9b2" candidate_uuid: "64769abce709430683bbb28596200d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4992d2f13b0744c690f509ad55cff4e9" is_pre_election: true
I20260620 18:37:05.155220 30250 raft_consensus.cc:2454] T f6aebc4627a54ef594ab500b5ee7e9b2 P 4992d2f13b0744c690f509ad55cff4e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64769abce709430683bbb28596200d30 in term 0.
I20260620 18:37:05.155628 30578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:05.145787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:53455 (local address 127.10.131.2:39441)
0620 18:37:05.146022 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:05.146031 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:05.148798 (+ 2767us) server_negotiation.cc:407] Connection header received
0620 18:37:05.149070 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:05.149077 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:05.149165 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:05.149230 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:37:05.150073 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:05.151098 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:05.153594 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:05.154726 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:05.154910 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:05.154976 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:05.155261 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 18:37:05.155380 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260620 18:37:05.155558 30249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6aebc4627a54ef594ab500b5ee7e9b2" candidate_uuid: "0f0e44a8ffcf4ea8bcd3d74084d3af46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4992d2f13b0744c690f509ad55cff4e9"
I20260620 18:37:05.155790 30592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:05.145285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:39441 (local address 127.10.131.3:53455)
0620 18:37:05.148691 (+ 3406us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:05.148710 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:05.148920 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:05.149420 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:05.149427 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:05.149454 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:37:05.149886 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:05.149901 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:05.151308 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:05.151313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:05.153403 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:05.153418 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:05.154928 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:05.154934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:05.155110 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:05.155140 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 18:37:05.155253 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 18:37:05.155386 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3294,"mutex_wait_us":30,"thread_start_us":3109,"threads_started":1}
I20260620 18:37:05.155776 30249 raft_consensus.cc:3037] T f6aebc4627a54ef594ab500b5ee7e9b2 P 4992d2f13b0744c690f509ad55cff4e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:05.156894 30451 leader_election.cc:302] T f6aebc4627a54ef594ab500b5ee7e9b2 P 64769abce709430683bbb28596200d30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4992d2f13b0744c690f509ad55cff4e9, 64769abce709430683bbb28596200d30; no voters:
I20260620 18:37:05.156757 30384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6aebc4627a54ef594ab500b5ee7e9b2" candidate_uuid: "64769abce709430683bbb28596200d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f0e44a8ffcf4ea8bcd3d74084d3af46" is_pre_election: true
I20260620 18:37:05.157236 30384 raft_consensus.cc:2379] T f6aebc4627a54ef594ab500b5ee7e9b2 P 0f0e44a8ffcf4ea8bcd3d74084d3af46 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 64769abce709430683bbb28596200d30 in current term 1: Already voted for candidate 0f0e44a8ffcf4ea8bcd3d74084d3af46 in this term.
I20260620 18:37:05.158278 30518 raft_consensus.cc:2454] T f6aebc4627a54ef594ab500b5ee7e9b2 P 64769abce709430683bbb28596200d30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f0e44a8ffcf4ea8bcd3d74084d3af46 in term 1.
I20260620 18:37:05.158521 30588 raft_consensus.cc:2741] T f6aebc4627a54ef594ab500b5ee7e9b2 P 64769abce709430683bbb28596200d30 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260620 18:37:05.158818 30317 leader_election.cc:302] T f6aebc4627a54ef594ab500b5ee7e9b2 P 0f0e44a8ffcf4ea8bcd3d74084d3af46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f0e44a8ffcf4ea8bcd3d74084d3af46, 64769abce709430683bbb28596200d30; no voters:
I20260620 18:37:05.158851 30249 raft_consensus.cc:2454] T f6aebc4627a54ef594ab500b5ee7e9b2 P 4992d2f13b0744c690f509ad55cff4e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f0e44a8ffcf4ea8bcd3d74084d3af46 in term 1.
I20260620 18:37:05.159092 30587 raft_consensus.cc:2781] T f6aebc4627a54ef594ab500b5ee7e9b2 P 0f0e44a8ffcf4ea8bcd3d74084d3af46 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:37:05.159547 30587 raft_consensus.cc:686] T f6aebc4627a54ef594ab500b5ee7e9b2 P 0f0e44a8ffcf4ea8bcd3d74084d3af46 [term 1 LEADER]: Becoming Leader. State: Replica: 0f0e44a8ffcf4ea8bcd3d74084d3af46, State: Running, Role: LEADER
I20260620 18:37:05.160006 30587 consensus_queue.cc:237] T f6aebc4627a54ef594ab500b5ee7e9b2 P 0f0e44a8ffcf4ea8bcd3d74084d3af46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64769abce709430683bbb28596200d30" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 41103 } } peers { permanent_uuid: "4992d2f13b0744c690f509ad55cff4e9" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41599 } } peers { permanent_uuid: "0f0e44a8ffcf4ea8bcd3d74084d3af46" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39441 } }
I20260620 18:37:05.164834 30105 catalog_manager.cc:5495] T f6aebc4627a54ef594ab500b5ee7e9b2 P 0f0e44a8ffcf4ea8bcd3d74084d3af46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f0e44a8ffcf4ea8bcd3d74084d3af46 (127.10.131.2). New cstate: current_term: 1 leader_uuid: "0f0e44a8ffcf4ea8bcd3d74084d3af46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64769abce709430683bbb28596200d30" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 41103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4992d2f13b0744c690f509ad55cff4e9" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f0e44a8ffcf4ea8bcd3d74084d3af46" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 39441 } health_report { overall_health: HEALTHY } } }
I20260620 18:37:05.196254 30578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:05.191129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43976 (local address 127.10.131.2:39441)
0620 18:37:05.191304 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:05.191311 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:05.191459 (+ 148us) server_negotiation.cc:407] Connection header received
0620 18:37:05.191625 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:05.191629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:05.191707 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:05.191884 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:37:05.192181 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:05.192947 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:05.194260 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:05.194674 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:05.194794 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:05.195200 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:37:05.195686 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:37:05.195810 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:05.195871 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 18:37:05.195944 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 18:37:05.204543 30249 raft_consensus.cc:1261] T f6aebc4627a54ef594ab500b5ee7e9b2 P 4992d2f13b0744c690f509ad55cff4e9 [term 1 FOLLOWER]: Refusing update from remote peer 0f0e44a8ffcf4ea8bcd3d74084d3af46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:37:05.204543 30518 raft_consensus.cc:1261] T f6aebc4627a54ef594ab500b5ee7e9b2 P 64769abce709430683bbb28596200d30 [term 1 FOLLOWER]: Refusing update from remote peer 0f0e44a8ffcf4ea8bcd3d74084d3af46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:37:05.205427 30594 consensus_queue.cc:1035] T f6aebc4627a54ef594ab500b5ee7e9b2 P 0f0e44a8ffcf4ea8bcd3d74084d3af46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64769abce709430683bbb28596200d30" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 41103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:37:05.205796 30587 consensus_queue.cc:1035] T f6aebc4627a54ef594ab500b5ee7e9b2 P 0f0e44a8ffcf4ea8bcd3d74084d3af46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4992d2f13b0744c690f509ad55cff4e9" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 41599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:37:05.217628 30599 mvcc.cc:205] Tried to move back new op lower bound from 7298992640828301312 to 7298992640659812352. Current Snapshot: MvccSnapshot[applied={T|T < 7298992640828301312}]
I20260620 18:37:05.231096 30105 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:37:05.231374 30105 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:37:05.234722 30105 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a2b9515eba149bcbaf3d4e7f5b4c994: Sending DeleteTablet for 3 replicas of tablet f6aebc4627a54ef594ab500b5ee7e9b2
I20260620 18:37:05.235817 30364 tablet_service.cc:1514] Processing DeleteTablet for tablet f6aebc4627a54ef594ab500b5ee7e9b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:37:05 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:43974
I20260620 18:37:05.235893 30230 tablet_service.cc:1514] Processing DeleteTablet for tablet f6aebc4627a54ef594ab500b5ee7e9b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:37:05 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:45428
I20260620 18:37:05.236089 30498 tablet_service.cc:1514] Processing DeleteTablet for tablet f6aebc4627a54ef594ab500b5ee7e9b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:37:05 UTC) from {username='kudu', principal='kudu/127.10.131.62@KRBTEST.COM'} at 127.0.0.1:33762
I20260620 18:37:05.237905 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 30167
I20260620 18:37:05.245922 30609 tablet_replica.cc:330] T f6aebc4627a54ef594ab500b5ee7e9b2 P 64769abce709430683bbb28596200d30: stopping tablet replica
I20260620 18:37:05.247223 30609 raft_consensus.cc:2229] T f6aebc4627a54ef594ab500b5ee7e9b2 P 64769abce709430683bbb28596200d30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:37:05.248447 30609 raft_consensus.cc:2258] T f6aebc4627a54ef594ab500b5ee7e9b2 P 64769abce709430683bbb28596200d30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:37:05.248543 30607 tablet_replica.cc:330] T f6aebc4627a54ef594ab500b5ee7e9b2 P 0f0e44a8ffcf4ea8bcd3d74084d3af46: stopping tablet replica
I20260620 18:37:05.248937 30607 raft_consensus.cc:2229] T f6aebc4627a54ef594ab500b5ee7e9b2 P 0f0e44a8ffcf4ea8bcd3d74084d3af46 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:37:05.249495 30607 raft_consensus.cc:2258] T f6aebc4627a54ef594ab500b5ee7e9b2 P 0f0e44a8ffcf4ea8bcd3d74084d3af46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:37:05.250952 30609 ts_tablet_manager.cc:1905] T f6aebc4627a54ef594ab500b5ee7e9b2 P 64769abce709430683bbb28596200d30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:37:05.254499 30607 ts_tablet_manager.cc:1905] T f6aebc4627a54ef594ab500b5ee7e9b2 P 0f0e44a8ffcf4ea8bcd3d74084d3af46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:37:05.256951 30608 tablet_replica.cc:330] T f6aebc4627a54ef594ab500b5ee7e9b2 P 4992d2f13b0744c690f509ad55cff4e9: stopping tablet replica
I20260620 18:37:05.257350 30609 ts_tablet_manager.cc:1918] T f6aebc4627a54ef594ab500b5ee7e9b2 P 64769abce709430683bbb28596200d30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:37:05.257391 30608 raft_consensus.cc:2229] T f6aebc4627a54ef594ab500b5ee7e9b2 P 4992d2f13b0744c690f509ad55cff4e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:37:05.257550 30609 log.cc:1196] T f6aebc4627a54ef594ab500b5ee7e9b2 P 64769abce709430683bbb28596200d30: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-2/wal/wals/f6aebc4627a54ef594ab500b5ee7e9b2
Jun 20 18:37:05 dist-test-slave-cbn3 krb5kdc[30061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.1@KRBTEST.COM
I20260620 18:37:05.257920 30608 raft_consensus.cc:2258] T f6aebc4627a54ef594ab500b5ee7e9b2 P 4992d2f13b0744c690f509ad55cff4e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:37:05.258225 30609 ts_tablet_manager.cc:1939] T f6aebc4627a54ef594ab500b5ee7e9b2 P 64769abce709430683bbb28596200d30: Deleting consensus metadata
I20260620 18:37:05.260653 30090 catalog_manager.cc:4841] TS 64769abce709430683bbb28596200d30 (127.10.131.3:41103): tablet f6aebc4627a54ef594ab500b5ee7e9b2 (table test-table [id=0f967ad37cc7461082625fbef752491b]) successfully deleted
I20260620 18:37:05.263569 30607 ts_tablet_manager.cc:1918] T f6aebc4627a54ef594ab500b5ee7e9b2 P 0f0e44a8ffcf4ea8bcd3d74084d3af46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:37:05.264815 30608 ts_tablet_manager.cc:1905] T f6aebc4627a54ef594ab500b5ee7e9b2 P 4992d2f13b0744c690f509ad55cff4e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:37:05.266520 30580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:05.238992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45432 (local address 127.10.131.1:41599)
0620 18:37:05.239214 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:05.239223 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:05.245997 (+ 6774us) server_negotiation.cc:407] Connection header received
0620 18:37:05.246267 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:05.246273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:05.246369 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:05.246435 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:05.249828 (+ 3393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:05.250923 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:05.252779 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:05.253432 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:05.253564 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:05.259121 (+ 5557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:05.259134 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:05.259150 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:05.259189 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:05.263040 (+ 3851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:05.263898 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:05.263908 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:05.263911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:05.264074 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:05.264516 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:05.264523 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:05.264526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:05.264988 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:05.265292 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:05.265878 (+ 586us) server_negotiation.cc:299] Negotiation successful
0620 18:37:05.266160 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:37:05.268333 30290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:05.269259 30607 log.cc:1196] T f6aebc4627a54ef594ab500b5ee7e9b2 P 0f0e44a8ffcf4ea8bcd3d74084d3af46: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781980314095189-10764-0/minicluster-data/ts-1/wal/wals/f6aebc4627a54ef594ab500b5ee7e9b2
I20260620 18:37:05.269927 30607 ts_tablet_manager.cc:1939] T f6aebc4627a54ef594ab500b5ee7e9b2 P 0f0e44a8ffcf4ea8bcd3d74084d3af46: Deleting consensus metadata
I20260620 18:37:05.271695 30092 catalog_manager.cc:4841] TS 0f0e44a8ffcf4ea8bcd3d74084d3af46 (127.10.131.2:39441): tablet f6aebc4627a54ef594ab500b5ee7e9b2 (table test-table [id=0f967ad37cc7461082625fbef752491b]) successfully deleted
I20260620 18:37:05.360647 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 30167
I20260620 18:37:05.378614 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 30301
W20260620 18:37:05.378623 30090 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.131.1:41599 (error 108)
I20260620 18:37:05.382579 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:05.381637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:41599 (local address 127.0.0.1:45434)
0620 18:37:05.382154 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:05.382400 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:41599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339}
W20260620 18:37:05.383493 30090 catalog_manager.cc:4584] TS 4992d2f13b0744c690f509ad55cff4e9 (127.10.131.1:41599): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f6aebc4627a54ef594ab500b5ee7e9b2: Network error: Client connection negotiation failed: client connection to 127.10.131.1:41599: connect: Connection refused (error 111)
Jun 20 18:37:05 dist-test-slave-cbn3 krb5kdc[30061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.2@KRBTEST.COM
I20260620 18:37:05.397212 30578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:05.379727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43980 (local address 127.10.131.2:39441)
0620 18:37:05.380673 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:05.380681 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:05.380700 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:37:05.380766 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:05.380771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:05.380888 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:05.380988 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:05.383370 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:05.384344 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:05.385813 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:05.386278 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:05.386355 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:05.391007 (+ 4652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:05.391017 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:05.391029 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:05.391064 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:05.394359 (+ 3295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:05.395011 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:05.395026 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:05.395028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:05.395171 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:05.395597 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:05.395603 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:05.395606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:05.395979 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:05.396309 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:05.396609 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 18:37:05.396867 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715}
I20260620 18:37:05.398133 30424 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:05.437258 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:05.436685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:41599 (local address 127.0.0.1:45448)
0620 18:37:05.436938 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:05.437044 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:41599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 18:37:05.438331 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:05.437923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:41599 (local address 127.0.0.1:45452)
0620 18:37:05.438087 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:05.438186 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:41599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 18:37:05.498868 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:05.498344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:41599 (local address 127.0.0.1:45466)
0620 18:37:05.498577 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:05.498689 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:41599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 18:37:05.499912 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:05.499515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:41599 (local address 127.0.0.1:45476)
0620 18:37:05.499699 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:05.499781 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:41599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 18:37:05.510608 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 30301
I20260620 18:37:05.530320 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 30435
Jun 20 18:37:05 dist-test-slave-cbn3 krb5kdc[30061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.3@KRBTEST.COM
I20260620 18:37:05.544843 30576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:05.530991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33776 (local address 127.10.131.3:41103)
0620 18:37:05.531176 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:05.531183 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:05.531248 (+ 65us) server_negotiation.cc:407] Connection header received
0620 18:37:05.531454 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:05.531463 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:05.531528 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:05.531628 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:05.532879 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:05.533660 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:05.534939 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:05.535328 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:05.535455 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:05.539977 (+ 4522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:05.539991 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:05.540007 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:05.540044 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:05.542631 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:05.543230 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:05.543237 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:05.543239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:05.543361 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:05.543663 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:05.543666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:05.543668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:05.543987 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:05.544187 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:05.544442 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 18:37:05.544595 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:37:05.545526 30558 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:05.580390 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:05.579819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:41599 (local address 127.0.0.1:45480)
0620 18:37:05.580056 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:05.580178 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:41599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 18:37:05.581456 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:05.581024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:41599 (local address 127.0.0.1:45496)
0620 18:37:05.581248 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:05.581330 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:41599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 18:37:05.638830 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 30435
I20260620 18:37:05.655540 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 30074
I20260620 18:37:05.656540 30136 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:05.765463 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 30074
I20260620 18:37:05.765491 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:05.764823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:41599 (local address 127.0.0.1:45508)
0620 18:37:05.765156 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:05.765302 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:41599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
2026-06-20T18:37:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4344 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T18:37:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:37:05Z Disabled control of system clock
I20260620 18:37:05.842224 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:45919
--webserver_interface=127.10.131.62
--webserver_port=0
--builtin_ntp_servers=127.10.131.20:44747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:45919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:37:06.167865 30619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:06.168135 30619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:06.168222 30619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:06.175480 30619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:37:06.175583 30619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:06.175621 30619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:06.175678 30619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:37:06.175721 30619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:37:06.184803 30619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:44747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:45919
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:45919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:37:06.186039 30619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:06.187726 30619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:06.199684 30624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:06.199604 30625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:06.199651 30627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:06.200495 30619 server_base.cc:1034] running on GCE node
I20260620 18:37:06.201324 30619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:06.202695 30619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:06.203890 30619 hybrid_clock.cc:648] HybridClock initialized: now 1781980626203853 us; error 26 us; skew 500 ppm
I20260620 18:37:06.204267 30619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:37:06.206528 30619 webserver.cc:460] Webserver started at http://127.10.131.62:34215/ using document root <none> and password file <none>
I20260620 18:37:06.207269 30619 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:06.207382 30619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:06.207715 30619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:06.210593 30619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "812daa8a136745809f1c4152f398337f"
format_stamp: "Formatted at 2026-06-20 18:37:06 on dist-test-slave-cbn3"
server_key: "0c9acd9e24d42fd1a48641d4e0355c61"
server_key_iv: "e3080989fc08a6ea1050c88a51067497"
server_key_version: "encryptionkey@0"
I20260620 18:37:06.211393 30619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "812daa8a136745809f1c4152f398337f"
format_stamp: "Formatted at 2026-06-20 18:37:06 on dist-test-slave-cbn3"
server_key: "0c9acd9e24d42fd1a48641d4e0355c61"
server_key_iv: "e3080989fc08a6ea1050c88a51067497"
server_key_version: "encryptionkey@0"
I20260620 18:37:06.216290 30619 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 18:37:06.219697 30633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:06.221063 30619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:37:06.221252 30619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "812daa8a136745809f1c4152f398337f"
format_stamp: "Formatted at 2026-06-20 18:37:06 on dist-test-slave-cbn3"
server_key: "0c9acd9e24d42fd1a48641d4e0355c61"
server_key_iv: "e3080989fc08a6ea1050c88a51067497"
server_key_version: "encryptionkey@0"
I20260620 18:37:06.221458 30619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:06.239661 30619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:06.240489 30619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:06.240775 30619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:06.258292 30619 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:45919
I20260620 18:37:06.258315 30684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:45919 every 8 connection(s)
I20260620 18:37:06.259814 30619 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:37:06.263321 30685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:06.264309 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 30619
I20260620 18:37:06.264480 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:37:06.265048 10764 external_mini_cluster.cc:1428] Setting key 26b0e7b40efe05fb8eac6bfeca1f764b
I20260620 18:37:06.275561 30685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 812daa8a136745809f1c4152f398337f: Bootstrap starting.
I20260620 18:37:06.277880 30688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:06.266846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44386 (local address 127.10.131.62:45919)
0620 18:37:06.267820 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:06.267831 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:06.267875 (+ 44us) server_negotiation.cc:407] Connection header received
0620 18:37:06.269731 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:06.269751 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:06.270200 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:06.270581 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:06.271099 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:06.272335 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:06.273739 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:06.274387 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:06.274576 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:06.275766 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:06.275776 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:06.275799 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:06.276306 (+ 507us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:06.276359 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:06.276383 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:06.276508 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:06.276703 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 18:37:06.277015 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":381,"threads_started":1}
I20260620 18:37:06.278954 30685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 812daa8a136745809f1c4152f398337f: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:06.279938 30685 log.cc:824] T 00000000000000000000000000000000 P 812daa8a136745809f1c4152f398337f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:06.282565 30685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 812daa8a136745809f1c4152f398337f: No bootstrap required, opened a new log
I20260620 18:37:06.289865 30685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 812daa8a136745809f1c4152f398337f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "812daa8a136745809f1c4152f398337f" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45919 } }
I20260620 18:37:06.290272 30685 raft_consensus.cc:374] T 00000000000000000000000000000000 P 812daa8a136745809f1c4152f398337f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:06.290386 30685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 812daa8a136745809f1c4152f398337f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 812daa8a136745809f1c4152f398337f, State: Initialized, Role: FOLLOWER
I20260620 18:37:06.291023 30685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 812daa8a136745809f1c4152f398337f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "812daa8a136745809f1c4152f398337f" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45919 } }
I20260620 18:37:06.291263 30685 raft_consensus.cc:388] T 00000000000000000000000000000000 P 812daa8a136745809f1c4152f398337f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:37:06.291410 30685 raft_consensus.cc:482] T 00000000000000000000000000000000 P 812daa8a136745809f1c4152f398337f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:37:06.291586 30685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 812daa8a136745809f1c4152f398337f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:06.294013 30685 raft_consensus.cc:504] T 00000000000000000000000000000000 P 812daa8a136745809f1c4152f398337f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "812daa8a136745809f1c4152f398337f" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45919 } }
I20260620 18:37:06.294597 30685 leader_election.cc:302] T 00000000000000000000000000000000 P 812daa8a136745809f1c4152f398337f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 812daa8a136745809f1c4152f398337f; no voters:
I20260620 18:37:06.295312 30685 leader_election.cc:288] T 00000000000000000000000000000000 P 812daa8a136745809f1c4152f398337f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:37:06.295485 30690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 812daa8a136745809f1c4152f398337f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:37:06.297076 30690 raft_consensus.cc:686] T 00000000000000000000000000000000 P 812daa8a136745809f1c4152f398337f [term 1 LEADER]: Becoming Leader. State: Replica: 812daa8a136745809f1c4152f398337f, State: Running, Role: LEADER
I20260620 18:37:06.297554 30685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 812daa8a136745809f1c4152f398337f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:37:06.297609 30690 consensus_queue.cc:237] T 00000000000000000000000000000000 P 812daa8a136745809f1c4152f398337f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "812daa8a136745809f1c4152f398337f" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45919 } }
I20260620 18:37:06.300239 30692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 812daa8a136745809f1c4152f398337f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 812daa8a136745809f1c4152f398337f. Latest consensus state: current_term: 1 leader_uuid: "812daa8a136745809f1c4152f398337f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "812daa8a136745809f1c4152f398337f" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45919 } } }
I20260620 18:37:06.300634 30692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 812daa8a136745809f1c4152f398337f [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:06.301209 30691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 812daa8a136745809f1c4152f398337f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "812daa8a136745809f1c4152f398337f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "812daa8a136745809f1c4152f398337f" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45919 } } }
I20260620 18:37:06.301548 30691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 812daa8a136745809f1c4152f398337f [sys.catalog]: This master's current role is: LEADER
W20260620 18:37:06.310956 30707 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 812daa8a136745809f1c4152f398337f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:37:06.311064 30707 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:37:06.311210 30699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:37:06.313452 30699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:37:06.321637 30699 catalog_manager.cc:1347] Generated new cluster ID: eb3ffbb11d7e427399bf93b3236530f9
I20260620 18:37:06.321748 30699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:37:06.353971 30699 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:37:06.355222 30699 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:37:06.360574 30699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 812daa8a136745809f1c4152f398337f: Generated new TSK 0
I20260620 18:37:06.361608 30699 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:37:06.375385 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.1:0
--local_ip_for_outbound_sockets=127.10.131.1
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:45919
--builtin_ntp_servers=127.10.131.20:44747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:37:06.695590 30710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:06.695843 30710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:06.696003 30710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:06.703456 30710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:06.703547 30710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:06.703748 30710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.1
I20260620 18:37:06.712859 30710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:44747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.131.1
--webserver_port=0
--tserver_master_addrs=127.10.131.62:45919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.1
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:37:06.714049 30710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:06.715746 30710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:06.727970 30715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:06.728011 30716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:06.728046 30718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:06.728966 30710 server_base.cc:1034] running on GCE node
I20260620 18:37:06.729612 30710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:06.730497 30710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:06.731719 30710 hybrid_clock.cc:648] HybridClock initialized: now 1781980626731673 us; error 31 us; skew 500 ppm
I20260620 18:37:06.732023 30710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:37:06.734284 30710 webserver.cc:460] Webserver started at http://127.10.131.1:41015/ using document root <none> and password file <none>
I20260620 18:37:06.734951 30710 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:06.735080 30710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:06.735390 30710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:06.738212 30710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-0/data/instance:
uuid: "d572480e18e1449f90ab942dbace6fcf"
format_stamp: "Formatted at 2026-06-20 18:37:06 on dist-test-slave-cbn3"
server_key: "b1547d597ff6f27c021ef4840295d754"
server_key_iv: "4823a3e11e05198396fc730989a641c5"
server_key_version: "encryptionkey@0"
I20260620 18:37:06.738981 30710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance:
uuid: "d572480e18e1449f90ab942dbace6fcf"
format_stamp: "Formatted at 2026-06-20 18:37:06 on dist-test-slave-cbn3"
server_key: "b1547d597ff6f27c021ef4840295d754"
server_key_iv: "4823a3e11e05198396fc730989a641c5"
server_key_version: "encryptionkey@0"
I20260620 18:37:06.744340 30710 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:37:06.748034 30724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:06.749439 30710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:37:06.749615 30710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-0/wal
uuid: "d572480e18e1449f90ab942dbace6fcf"
format_stamp: "Formatted at 2026-06-20 18:37:06 on dist-test-slave-cbn3"
server_key: "b1547d597ff6f27c021ef4840295d754"
server_key_iv: "4823a3e11e05198396fc730989a641c5"
server_key_version: "encryptionkey@0"
I20260620 18:37:06.749837 30710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:06.780752 30710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:06.781728 30710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:06.782025 30710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:06.782953 30710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:37:06.784368 30710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:37:06.784494 30710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:06.784619 30710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:37:06.784709 30710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:06.817444 30710 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.1:40719
I20260620 18:37:06.817530 30836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.1:40719 every 8 connection(s)
I20260620 18:37:06.818727 30710 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-0/data/info.pb
I20260620 18:37:06.822252 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 30710
I20260620 18:37:06.822440 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-0/wal/instance
I20260620 18:37:06.822970 10764 external_mini_cluster.cc:1428] Setting key 9b7e577355dcd8562834deae28bffd7e
I20260620 18:37:06.825850 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.2:0
--local_ip_for_outbound_sockets=127.10.131.2
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:45919
--builtin_ntp_servers=127.10.131.20:44747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:37:06.831718 30839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:06.821284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:59169 (local address 127.10.131.62:45919)
0620 18:37:06.821739 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:06.821743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:06.822980 (+ 1237us) server_negotiation.cc:407] Connection header received
0620 18:37:06.825281 (+ 2301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:06.825287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:06.825375 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:06.825489 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:06.826659 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:06.827413 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:06.828959 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:06.829367 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:06.829504 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:06.830895 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:06.830900 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:06.830903 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:06.830920 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:06.830957 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:06.830969 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:06.831063 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:06.831278 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 18:37:06.831376 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":250,"threads_started":1}
I20260620 18:37:06.832680 30840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:06.821795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:45919 (local address 127.10.131.1:59169)
0620 18:37:06.822835 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:06.822871 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:06.825030 (+ 2159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:06.825730 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:06.825738 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:06.825851 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:06.826501 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:06.826517 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:06.827557 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:06.827563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:06.828794 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:06.828805 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:06.829587 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:06.829597 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:06.829841 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:06.830578 (+ 737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:37:06.830580 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:06.830620 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:37:06.830701 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:06.831061 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:06.831073 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:06.831083 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 18:37:06.831269 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 18:37:06.831648 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":313,"threads_started":1}
I20260620 18:37:06.834214 30837 heartbeater.cc:347] Connected to a master server at 127.10.131.62:45919
I20260620 18:37:06.834512 30837 heartbeater.cc:464] Registering TS with master...
I20260620 18:37:06.835366 30837 heartbeater.cc:511] Master 127.10.131.62:45919 requested a full tablet report, sending...
I20260620 18:37:06.837613 30650 ts_manager.cc:194] Registered new tserver with Master: d572480e18e1449f90ab942dbace6fcf (127.10.131.1:40719)
I20260620 18:37:06.839766 30650 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.131.1:59169
W20260620 18:37:07.153776 30841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:07.154102 30841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:07.154194 30841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:07.161644 30841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:07.161743 30841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:07.161850 30841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.2
I20260620 18:37:07.171000 30841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:44747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.131.2
--webserver_port=0
--tserver_master_addrs=127.10.131.62:45919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.2
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:37:07.172344 30841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:07.174021 30841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:07.187142 30846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:07.187103 30849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:07.187186 30847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:07.187649 30841 server_base.cc:1034] running on GCE node
I20260620 18:37:07.188406 30841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:07.189371 30841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:07.190601 30841 hybrid_clock.cc:648] HybridClock initialized: now 1781980627190552 us; error 66 us; skew 500 ppm
I20260620 18:37:07.190908 30841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:37:07.193166 30841 webserver.cc:460] Webserver started at http://127.10.131.2:39925/ using document root <none> and password file <none>
I20260620 18:37:07.193818 30841 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:07.193946 30841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:07.194257 30841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:07.196992 30841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-1/data/instance:
uuid: "355a3749b7c441dc9ca5e5ba546bcf40"
format_stamp: "Formatted at 2026-06-20 18:37:07 on dist-test-slave-cbn3"
server_key: "d4cff619092826e144daebce8ad463fe"
server_key_iv: "465d9733c7328b0c9d8b62cb6f6d8c2f"
server_key_version: "encryptionkey@0"
I20260620 18:37:07.197754 30841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance:
uuid: "355a3749b7c441dc9ca5e5ba546bcf40"
format_stamp: "Formatted at 2026-06-20 18:37:07 on dist-test-slave-cbn3"
server_key: "d4cff619092826e144daebce8ad463fe"
server_key_iv: "465d9733c7328b0c9d8b62cb6f6d8c2f"
server_key_version: "encryptionkey@0"
I20260620 18:37:07.202704 30841 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:37:07.205927 30855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:07.207295 30841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:37:07.207459 30841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-1/wal
uuid: "355a3749b7c441dc9ca5e5ba546bcf40"
format_stamp: "Formatted at 2026-06-20 18:37:07 on dist-test-slave-cbn3"
server_key: "d4cff619092826e144daebce8ad463fe"
server_key_iv: "465d9733c7328b0c9d8b62cb6f6d8c2f"
server_key_version: "encryptionkey@0"
I20260620 18:37:07.207666 30841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:07.229513 30841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:07.230322 30841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:07.230641 30841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:07.231568 30841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:37:07.232998 30841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:37:07.233083 30841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:07.233245 30841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:37:07.233330 30841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:07.265457 30841 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.2:41017
I20260620 18:37:07.265499 30967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.2:41017 every 8 connection(s)
I20260620 18:37:07.266983 30841 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-1/data/info.pb
I20260620 18:37:07.273226 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 30841
I20260620 18:37:07.273420 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-1/wal/instance
I20260620 18:37:07.274017 10764 external_mini_cluster.cc:1428] Setting key fee5dc3323020ccb6ef0c1e4a0fe49d4
I20260620 18:37:07.276997 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.131.3:0
--local_ip_for_outbound_sockets=127.10.131.3
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:45919
--builtin_ntp_servers=127.10.131.20:44747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:37:07.281675 30839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:07.269836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.2:36271 (local address 127.10.131.62:45919)
0620 18:37:07.270013 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:07.270018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:07.271472 (+ 1454us) server_negotiation.cc:407] Connection header received
0620 18:37:07.273528 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:07.273535 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:07.273615 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:07.273728 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:07.275009 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:07.276172 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:07.278662 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:07.279220 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:07.279317 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:07.280970 (+ 1653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:07.280975 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:07.280978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:07.280996 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:07.281024 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:07.281036 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:07.281106 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:07.281368 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 18:37:07.281444 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:37:07.283079 30970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:07.270428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:45919 (local address 127.10.131.2:36271)
0620 18:37:07.271321 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:07.271354 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:07.273165 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:07.273981 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:07.273993 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:07.274107 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:07.274823 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:07.274849 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:07.276339 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:07.276345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:07.278495 (+ 2150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:07.278513 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:07.279447 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:07.279457 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:07.279789 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:07.280545 (+ 756us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:37:07.280551 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:07.280598 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:37:07.280724 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:07.281138 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:07.281150 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:07.281161 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 18:37:07.281349 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 18:37:07.281759 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":292,"threads_started":1}
I20260620 18:37:07.284461 30968 heartbeater.cc:347] Connected to a master server at 127.10.131.62:45919
I20260620 18:37:07.284767 30968 heartbeater.cc:464] Registering TS with master...
I20260620 18:37:07.285491 30968 heartbeater.cc:511] Master 127.10.131.62:45919 requested a full tablet report, sending...
I20260620 18:37:07.286897 30650 ts_manager.cc:194] Registered new tserver with Master: 355a3749b7c441dc9ca5e5ba546bcf40 (127.10.131.2:41017)
I20260620 18:37:07.288089 30650 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.131.2:36271
W20260620 18:37:07.600337 30971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:07.600659 30971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:07.600778 30971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:07.608670 30971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:07.608795 30971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:07.609028 30971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.131.3
I20260620 18:37:07.621168 30971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:44747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.131.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.131.3
--webserver_port=0
--tserver_master_addrs=127.10.131.62:45919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.131.3
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:37:07.622337 30971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:07.624001 30971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:07.637367 30976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:07.637449 30979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:07.637472 30977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:07.638213 30971 server_base.cc:1034] running on GCE node
I20260620 18:37:07.638721 30971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:07.639555 30971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:07.640760 30971 hybrid_clock.cc:648] HybridClock initialized: now 1781980627640724 us; error 31 us; skew 500 ppm
I20260620 18:37:07.641094 30971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:37:07.643142 30971 webserver.cc:460] Webserver started at http://127.10.131.3:35237/ using document root <none> and password file <none>
I20260620 18:37:07.643783 30971 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:07.643862 30971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:07.644096 30971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:07.646965 30971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-2/data/instance:
uuid: "c5245d05ac0e49f7984122af9b29895e"
format_stamp: "Formatted at 2026-06-20 18:37:07 on dist-test-slave-cbn3"
server_key: "1467a4e2b44adb811c7e607c75442cdf"
server_key_iv: "7ee5c5020ef786ff241ac135b20ecf35"
server_key_version: "encryptionkey@0"
I20260620 18:37:07.647694 30971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance:
uuid: "c5245d05ac0e49f7984122af9b29895e"
format_stamp: "Formatted at 2026-06-20 18:37:07 on dist-test-slave-cbn3"
server_key: "1467a4e2b44adb811c7e607c75442cdf"
server_key_iv: "7ee5c5020ef786ff241ac135b20ecf35"
server_key_version: "encryptionkey@0"
I20260620 18:37:07.652627 30971 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:37:07.656175 30985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:07.657549 30971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:37:07.657684 30971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-2/wal
uuid: "c5245d05ac0e49f7984122af9b29895e"
format_stamp: "Formatted at 2026-06-20 18:37:07 on dist-test-slave-cbn3"
server_key: "1467a4e2b44adb811c7e607c75442cdf"
server_key_iv: "7ee5c5020ef786ff241ac135b20ecf35"
server_key_version: "encryptionkey@0"
I20260620 18:37:07.657858 30971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:07.691082 30971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:07.691901 30971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:07.692163 30971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:07.693156 30971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:37:07.694625 30971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:37:07.694708 30971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:07.694777 30971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:37:07.694832 30971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:07.728003 30971 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.3:44315
I20260620 18:37:07.728106 31097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.3:44315 every 8 connection(s)
I20260620 18:37:07.729475 30971 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-2/data/info.pb
I20260620 18:37:07.735224 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 30971
I20260620 18:37:07.735424 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-2/wal/instance
I20260620 18:37:07.736001 10764 external_mini_cluster.cc:1428] Setting key 3e4d8ec89e60f1ab36544a565f6e06f5
I20260620 18:37:07.741982 30839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:07.731819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.3:39439 (local address 127.10.131.62:45919)
0620 18:37:07.732050 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:07.732057 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:07.733412 (+ 1355us) server_negotiation.cc:407] Connection header received
0620 18:37:07.735292 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:07.735323 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:07.735418 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:07.735548 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:07.736835 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:07.737756 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:07.739273 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:07.739681 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:07.739820 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:07.741196 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:07.741201 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:07.741204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:07.741221 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:07.741251 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:07.741263 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:07.741365 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:07.741579 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 18:37:07.741673 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 18:37:07.742939 31100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:07.732382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:45919 (local address 127.10.131.3:39439)
0620 18:37:07.733265 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:07.733298 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:07.734995 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:07.735824 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:07.735838 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:07.735947 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:07.736663 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:07.736682 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:07.737916 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:07.737926 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:07.739118 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:07.739129 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:07.739897 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:07.739903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:07.740123 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:07.740810 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:37:07.740813 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:07.740897 (+ 84us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:37:07.740985 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:07.741363 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:07.741374 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:07.741383 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 18:37:07.741569 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 18:37:07.741933 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":255,"threads_started":1}
I20260620 18:37:07.744212 31098 heartbeater.cc:347] Connected to a master server at 127.10.131.62:45919
I20260620 18:37:07.744495 31098 heartbeater.cc:464] Registering TS with master...
I20260620 18:37:07.745239 31098 heartbeater.cc:511] Master 127.10.131.62:45919 requested a full tablet report, sending...
I20260620 18:37:07.746542 30650 ts_manager.cc:194] Registered new tserver with Master: c5245d05ac0e49f7984122af9b29895e (127.10.131.3:44315)
I20260620 18:37:07.747308 30650 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.131.3:39439
I20260620 18:37:07.750869 10764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:37:07.760067 30839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:07.754932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44392 (local address 127.10.131.62:45919)
0620 18:37:07.755093 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:07.755098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:07.755499 (+ 401us) server_negotiation.cc:407] Connection header received
0620 18:37:07.755626 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:07.755630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:07.755697 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:07.755774 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:07.756178 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:07.756825 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:07.758130 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:07.758500 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:07.758588 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:07.759525 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:07.759529 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:07.759532 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:07.759549 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:07.759578 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:07.759589 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:07.759656 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:07.759787 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 18:37:07.759853 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 18:37:07.764088 30650 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:37:07.770711 30650 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:37:07.816725 31114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:07.782109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:41017 (local address 127.0.0.1:43492)
0620 18:37:07.807781 (+ 25672us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:07.807820 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:07.807967 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:07.809330 (+ 1363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:07.809337 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:07.809380 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:37:07.809746 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:07.809757 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:07.811671 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:07.811677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:07.814024 (+ 2347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:07.814042 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:07.815738 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:07.815744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:07.815928 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:07.815981 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 18:37:07.816142 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 18:37:07.816334 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25419,"mutex_wait_us":24,"spinlock_wait_cycles":17792,"thread_start_us":24437,"threads_started":1}
I20260620 18:37:07.817262 31113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:07.782708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43492 (local address 127.10.131.2:41017)
0620 18:37:07.806185 (+ 23477us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:07.806196 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:07.808322 (+ 2126us) server_negotiation.cc:407] Connection header received
0620 18:37:07.808458 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:07.808469 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:07.808793 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:07.809035 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:37:07.809919 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:07.811538 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:07.814253 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:07.815550 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:07.815708 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:07.815951 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:07.816183 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 18:37:07.816346 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23289,"thread_start_us":23502,"threads_started":1}
I20260620 18:37:07.817829 31112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:07.781765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:40719 (local address 127.0.0.1:33720)
0620 18:37:07.806975 (+ 25210us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:07.807014 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:07.807208 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:07.810424 (+ 3216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:07.810428 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:07.810463 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:37:07.810804 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:07.811076 (+ 272us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:07.812923 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:07.812928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:07.815108 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:07.815122 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:07.817102 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:07.817117 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:07.817280 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:07.817320 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 18:37:07.817397 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 18:37:07.817521 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24949,"mutex_wait_us":58,"thread_start_us":27278,"threads_started":1}
I20260620 18:37:07.818058 31111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:07.782024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33720 (local address 127.10.131.1:40719)
0620 18:37:07.809657 (+ 27633us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:07.809667 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:07.809695 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:37:07.809771 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:07.809776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:07.809991 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:07.810195 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:37:07.811265 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:07.812746 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:07.815305 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:07.816622 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:07.816734 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:07.817487 (+ 753us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:07.817550 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 18:37:07.817700 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27504,"thread_start_us":27409,"threads_started":1}
I20260620 18:37:07.818929 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:07.782798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:44315 (local address 127.0.0.1:40752)
0620 18:37:07.809304 (+ 26506us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:07.809332 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:07.809448 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:07.810132 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:07.810136 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:07.810157 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:37:07.810507 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:07.810525 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:07.812436 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:07.812442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:07.814122 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:07.814137 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:07.816202 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:07.816209 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:07.816389 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:07.816423 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 18:37:07.816507 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 18:37:07.816643 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26388,"mutex_wait_us":39,"thread_start_us":23875,"threads_started":1}
I20260620 18:37:07.817337 31115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:07.783421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40752 (local address 127.10.131.3:44315)
0620 18:37:07.806309 (+ 22888us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:07.806316 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:07.809589 (+ 3273us) server_negotiation.cc:407] Connection header received
0620 18:37:07.809708 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:07.809716 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:07.809945 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:07.810128 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:37:07.810737 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:07.812293 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:07.814340 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:07.815721 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:07.815838 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:07.816081 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:07.816807 (+ 726us) server_negotiation.cc:299] Negotiation successful
0620 18:37:07.816990 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22737,"thread_start_us":22899,"threads_started":1}
I20260620 18:37:07.821210 30903 tablet_service.cc:1467] Processing CreateTablet for tablet c21635d0a02f475cac7e71b202f45662 (DEFAULT_TABLE table=test-table [id=126e0456b2b248feb21bff25856a91fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:37:07.822999 30903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c21635d0a02f475cac7e71b202f45662. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:07.822567 30772 tablet_service.cc:1467] Processing CreateTablet for tablet c21635d0a02f475cac7e71b202f45662 (DEFAULT_TABLE table=test-table [id=126e0456b2b248feb21bff25856a91fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:37:07.822948 31033 tablet_service.cc:1467] Processing CreateTablet for tablet c21635d0a02f475cac7e71b202f45662 (DEFAULT_TABLE table=test-table [id=126e0456b2b248feb21bff25856a91fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:37:07.824102 30772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c21635d0a02f475cac7e71b202f45662. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:07.824270 31033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c21635d0a02f475cac7e71b202f45662. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:07.835575 31116 tablet_bootstrap.cc:492] T c21635d0a02f475cac7e71b202f45662 P c5245d05ac0e49f7984122af9b29895e: Bootstrap starting.
I20260620 18:37:07.837577 31117 tablet_bootstrap.cc:492] T c21635d0a02f475cac7e71b202f45662 P 355a3749b7c441dc9ca5e5ba546bcf40: Bootstrap starting.
I20260620 18:37:07.840027 31116 tablet_bootstrap.cc:654] T c21635d0a02f475cac7e71b202f45662 P c5245d05ac0e49f7984122af9b29895e: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:07.841061 31117 tablet_bootstrap.cc:654] T c21635d0a02f475cac7e71b202f45662 P 355a3749b7c441dc9ca5e5ba546bcf40: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:07.841348 31116 log.cc:824] T c21635d0a02f475cac7e71b202f45662 P c5245d05ac0e49f7984122af9b29895e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:07.841727 31118 tablet_bootstrap.cc:492] T c21635d0a02f475cac7e71b202f45662 P d572480e18e1449f90ab942dbace6fcf: Bootstrap starting.
I20260620 18:37:07.842542 31117 log.cc:824] T c21635d0a02f475cac7e71b202f45662 P 355a3749b7c441dc9ca5e5ba546bcf40: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:07.843361 30837 heartbeater.cc:503] Master 127.10.131.62:45919 was elected leader, sending a full tablet report...
I20260620 18:37:07.844534 31116 tablet_bootstrap.cc:492] T c21635d0a02f475cac7e71b202f45662 P c5245d05ac0e49f7984122af9b29895e: No bootstrap required, opened a new log
I20260620 18:37:07.844785 31116 ts_tablet_manager.cc:1397] T c21635d0a02f475cac7e71b202f45662 P c5245d05ac0e49f7984122af9b29895e: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 18:37:07.845288 31118 tablet_bootstrap.cc:654] T c21635d0a02f475cac7e71b202f45662 P d572480e18e1449f90ab942dbace6fcf: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:07.846403 31118 log.cc:824] T c21635d0a02f475cac7e71b202f45662 P d572480e18e1449f90ab942dbace6fcf: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:07.846555 31117 tablet_bootstrap.cc:492] T c21635d0a02f475cac7e71b202f45662 P 355a3749b7c441dc9ca5e5ba546bcf40: No bootstrap required, opened a new log
I20260620 18:37:07.846810 31117 ts_tablet_manager.cc:1397] T c21635d0a02f475cac7e71b202f45662 P 355a3749b7c441dc9ca5e5ba546bcf40: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 18:37:07.850835 31118 tablet_bootstrap.cc:492] T c21635d0a02f475cac7e71b202f45662 P d572480e18e1449f90ab942dbace6fcf: No bootstrap required, opened a new log
I20260620 18:37:07.851063 31118 ts_tablet_manager.cc:1397] T c21635d0a02f475cac7e71b202f45662 P d572480e18e1449f90ab942dbace6fcf: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 18:37:07.855058 31117 raft_consensus.cc:348] T c21635d0a02f475cac7e71b202f45662 P 355a3749b7c441dc9ca5e5ba546bcf40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d572480e18e1449f90ab942dbace6fcf" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40719 } } peers { permanent_uuid: "355a3749b7c441dc9ca5e5ba546bcf40" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41017 } } peers { permanent_uuid: "c5245d05ac0e49f7984122af9b29895e" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 44315 } }
I20260620 18:37:07.855438 31117 raft_consensus.cc:374] T c21635d0a02f475cac7e71b202f45662 P 355a3749b7c441dc9ca5e5ba546bcf40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:07.855531 31117 raft_consensus.cc:729] T c21635d0a02f475cac7e71b202f45662 P 355a3749b7c441dc9ca5e5ba546bcf40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 355a3749b7c441dc9ca5e5ba546bcf40, State: Initialized, Role: FOLLOWER
I20260620 18:37:07.855322 31116 raft_consensus.cc:348] T c21635d0a02f475cac7e71b202f45662 P c5245d05ac0e49f7984122af9b29895e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d572480e18e1449f90ab942dbace6fcf" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40719 } } peers { permanent_uuid: "355a3749b7c441dc9ca5e5ba546bcf40" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41017 } } peers { permanent_uuid: "c5245d05ac0e49f7984122af9b29895e" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 44315 } }
I20260620 18:37:07.855700 31116 raft_consensus.cc:374] T c21635d0a02f475cac7e71b202f45662 P c5245d05ac0e49f7984122af9b29895e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:07.855787 31116 raft_consensus.cc:729] T c21635d0a02f475cac7e71b202f45662 P c5245d05ac0e49f7984122af9b29895e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5245d05ac0e49f7984122af9b29895e, State: Initialized, Role: FOLLOWER
I20260620 18:37:07.856231 31117 consensus_queue.cc:260] T c21635d0a02f475cac7e71b202f45662 P 355a3749b7c441dc9ca5e5ba546bcf40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d572480e18e1449f90ab942dbace6fcf" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40719 } } peers { permanent_uuid: "355a3749b7c441dc9ca5e5ba546bcf40" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41017 } } peers { permanent_uuid: "c5245d05ac0e49f7984122af9b29895e" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 44315 } }
I20260620 18:37:07.856338 31116 consensus_queue.cc:260] T c21635d0a02f475cac7e71b202f45662 P c5245d05ac0e49f7984122af9b29895e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d572480e18e1449f90ab942dbace6fcf" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40719 } } peers { permanent_uuid: "355a3749b7c441dc9ca5e5ba546bcf40" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41017 } } peers { permanent_uuid: "c5245d05ac0e49f7984122af9b29895e" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 44315 } }
I20260620 18:37:07.857273 30968 heartbeater.cc:503] Master 127.10.131.62:45919 was elected leader, sending a full tablet report...
I20260620 18:37:07.857972 31117 ts_tablet_manager.cc:1428] T c21635d0a02f475cac7e71b202f45662 P 355a3749b7c441dc9ca5e5ba546bcf40: Time spent starting tablet: real 0.011s user 0.000s sys 0.012s
I20260620 18:37:07.859285 31098 heartbeater.cc:503] Master 127.10.131.62:45919 was elected leader, sending a full tablet report...
I20260620 18:37:07.861043 31116 ts_tablet_manager.cc:1428] T c21635d0a02f475cac7e71b202f45662 P c5245d05ac0e49f7984122af9b29895e: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260620 18:37:07.861835 31118 raft_consensus.cc:348] T c21635d0a02f475cac7e71b202f45662 P d572480e18e1449f90ab942dbace6fcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d572480e18e1449f90ab942dbace6fcf" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40719 } } peers { permanent_uuid: "355a3749b7c441dc9ca5e5ba546bcf40" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41017 } } peers { permanent_uuid: "c5245d05ac0e49f7984122af9b29895e" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 44315 } }
I20260620 18:37:07.862253 31118 raft_consensus.cc:374] T c21635d0a02f475cac7e71b202f45662 P d572480e18e1449f90ab942dbace6fcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:07.862368 31118 raft_consensus.cc:729] T c21635d0a02f475cac7e71b202f45662 P d572480e18e1449f90ab942dbace6fcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d572480e18e1449f90ab942dbace6fcf, State: Initialized, Role: FOLLOWER
I20260620 18:37:07.862896 31118 consensus_queue.cc:260] T c21635d0a02f475cac7e71b202f45662 P d572480e18e1449f90ab942dbace6fcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d572480e18e1449f90ab942dbace6fcf" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40719 } } peers { permanent_uuid: "355a3749b7c441dc9ca5e5ba546bcf40" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41017 } } peers { permanent_uuid: "c5245d05ac0e49f7984122af9b29895e" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 44315 } }
I20260620 18:37:07.864411 31118 ts_tablet_manager.cc:1428] T c21635d0a02f475cac7e71b202f45662 P d572480e18e1449f90ab942dbace6fcf: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260620 18:37:07.919646 31124 raft_consensus.cc:482] T c21635d0a02f475cac7e71b202f45662 P d572480e18e1449f90ab942dbace6fcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:37:07.919984 31124 raft_consensus.cc:504] T c21635d0a02f475cac7e71b202f45662 P d572480e18e1449f90ab942dbace6fcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d572480e18e1449f90ab942dbace6fcf" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40719 } } peers { permanent_uuid: "355a3749b7c441dc9ca5e5ba546bcf40" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41017 } } peers { permanent_uuid: "c5245d05ac0e49f7984122af9b29895e" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 44315 } }
I20260620 18:37:07.921705 31124 leader_election.cc:288] T c21635d0a02f475cac7e71b202f45662 P d572480e18e1449f90ab942dbace6fcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 355a3749b7c441dc9ca5e5ba546bcf40 (127.10.131.2:41017), c5245d05ac0e49f7984122af9b29895e (127.10.131.3:44315)
I20260620 18:37:07.929139 31113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:07.921967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:49715 (local address 127.10.131.2:41017)
0620 18:37:07.922155 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:07.922164 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:07.922838 (+ 674us) server_negotiation.cc:407] Connection header received
0620 18:37:07.923072 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:07.923077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:07.923209 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:07.923302 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:37:07.923976 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:07.925156 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:07.927324 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:07.928084 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:07.928303 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:07.928376 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:07.928715 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 18:37:07.928870 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:37:07.929138 31115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:07.922278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.131.1:51183 (local address 127.10.131.3:44315)
0620 18:37:07.922487 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:07.922496 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:07.923140 (+ 644us) server_negotiation.cc:407] Connection header received
0620 18:37:07.923276 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:07.923284 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:07.923353 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:07.923435 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:37:07.924132 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:07.925260 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:07.927313 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:07.928093 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:07.928331 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:07.928401 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:07.928721 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 18:37:07.928882 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:37:07.929281 31126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:07.922155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.3:44315 (local address 127.10.131.1:51183)
0620 18:37:07.923028 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:07.923045 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:07.923155 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:07.923479 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:07.923485 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:07.923520 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:37:07.923961 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:07.923972 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:07.925411 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:07.925416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:07.927098 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:07.927120 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:07.928362 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:07.928369 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:07.928554 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:07.928576 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:37:07.928682 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 18:37:07.928884 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"mutex_wait_us":123,"spinlock_wait_cycles":16640,"thread_start_us":520,"threads_started":1}
I20260620 18:37:07.929659 31125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:07.921899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.2:41017 (local address 127.10.131.1:49715)
0620 18:37:07.922710 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:07.922724 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:07.922913 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:07.923326 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:07.923329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:07.923347 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:37:07.923809 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:07.923826 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:07.925339 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:07.925344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:07.927125 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:07.927137 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:07.928319 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:07.928325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:07.928491 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:07.928564 (+ 73us) client_negotiation.cc:769] Sending connection context
0620 18:37:07.928686 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 18:37:07.928880 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":106,"spinlock_wait_cycles":21760,"thread_start_us":500,"threads_started":1}
I20260620 18:37:07.930415 31053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c21635d0a02f475cac7e71b202f45662" candidate_uuid: "d572480e18e1449f90ab942dbace6fcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5245d05ac0e49f7984122af9b29895e" is_pre_election: true
I20260620 18:37:07.930480 30923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c21635d0a02f475cac7e71b202f45662" candidate_uuid: "d572480e18e1449f90ab942dbace6fcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "355a3749b7c441dc9ca5e5ba546bcf40" is_pre_election: true
I20260620 18:37:07.930943 31053 raft_consensus.cc:2454] T c21635d0a02f475cac7e71b202f45662 P c5245d05ac0e49f7984122af9b29895e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d572480e18e1449f90ab942dbace6fcf in term 0.
I20260620 18:37:07.930968 30923 raft_consensus.cc:2454] T c21635d0a02f475cac7e71b202f45662 P 355a3749b7c441dc9ca5e5ba546bcf40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d572480e18e1449f90ab942dbace6fcf in term 0.
I20260620 18:37:07.931653 30728 leader_election.cc:302] T c21635d0a02f475cac7e71b202f45662 P d572480e18e1449f90ab942dbace6fcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 355a3749b7c441dc9ca5e5ba546bcf40, d572480e18e1449f90ab942dbace6fcf; no voters:
I20260620 18:37:07.932072 31124 raft_consensus.cc:2781] T c21635d0a02f475cac7e71b202f45662 P d572480e18e1449f90ab942dbace6fcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:37:07.932228 31124 raft_consensus.cc:482] T c21635d0a02f475cac7e71b202f45662 P d572480e18e1449f90ab942dbace6fcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:37:07.932327 31124 raft_consensus.cc:3037] T c21635d0a02f475cac7e71b202f45662 P d572480e18e1449f90ab942dbace6fcf [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:07.934507 31124 raft_consensus.cc:504] T c21635d0a02f475cac7e71b202f45662 P d572480e18e1449f90ab942dbace6fcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d572480e18e1449f90ab942dbace6fcf" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40719 } } peers { permanent_uuid: "355a3749b7c441dc9ca5e5ba546bcf40" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41017 } } peers { permanent_uuid: "c5245d05ac0e49f7984122af9b29895e" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 44315 } }
I20260620 18:37:07.935192 31124 leader_election.cc:288] T c21635d0a02f475cac7e71b202f45662 P d572480e18e1449f90ab942dbace6fcf [CANDIDATE]: Term 1 election: Requested vote from peers 355a3749b7c441dc9ca5e5ba546bcf40 (127.10.131.2:41017), c5245d05ac0e49f7984122af9b29895e (127.10.131.3:44315)
I20260620 18:37:07.935616 30923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c21635d0a02f475cac7e71b202f45662" candidate_uuid: "d572480e18e1449f90ab942dbace6fcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "355a3749b7c441dc9ca5e5ba546bcf40"
I20260620 18:37:07.935696 31053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c21635d0a02f475cac7e71b202f45662" candidate_uuid: "d572480e18e1449f90ab942dbace6fcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5245d05ac0e49f7984122af9b29895e"
I20260620 18:37:07.935832 30923 raft_consensus.cc:3037] T c21635d0a02f475cac7e71b202f45662 P 355a3749b7c441dc9ca5e5ba546bcf40 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:07.935907 31053 raft_consensus.cc:3037] T c21635d0a02f475cac7e71b202f45662 P c5245d05ac0e49f7984122af9b29895e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:07.939126 30923 raft_consensus.cc:2454] T c21635d0a02f475cac7e71b202f45662 P 355a3749b7c441dc9ca5e5ba546bcf40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d572480e18e1449f90ab942dbace6fcf in term 1.
I20260620 18:37:07.939138 31053 raft_consensus.cc:2454] T c21635d0a02f475cac7e71b202f45662 P c5245d05ac0e49f7984122af9b29895e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d572480e18e1449f90ab942dbace6fcf in term 1.
I20260620 18:37:07.939589 30725 leader_election.cc:302] T c21635d0a02f475cac7e71b202f45662 P d572480e18e1449f90ab942dbace6fcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5245d05ac0e49f7984122af9b29895e, d572480e18e1449f90ab942dbace6fcf; no voters:
I20260620 18:37:07.939900 31124 raft_consensus.cc:2781] T c21635d0a02f475cac7e71b202f45662 P d572480e18e1449f90ab942dbace6fcf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:37:07.940311 31124 raft_consensus.cc:686] T c21635d0a02f475cac7e71b202f45662 P d572480e18e1449f90ab942dbace6fcf [term 1 LEADER]: Becoming Leader. State: Replica: d572480e18e1449f90ab942dbace6fcf, State: Running, Role: LEADER
I20260620 18:37:07.941099 31124 consensus_queue.cc:237] T c21635d0a02f475cac7e71b202f45662 P d572480e18e1449f90ab942dbace6fcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d572480e18e1449f90ab942dbace6fcf" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40719 } } peers { permanent_uuid: "355a3749b7c441dc9ca5e5ba546bcf40" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41017 } } peers { permanent_uuid: "c5245d05ac0e49f7984122af9b29895e" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 44315 } }
I20260620 18:37:07.945581 30649 catalog_manager.cc:5495] T c21635d0a02f475cac7e71b202f45662 P d572480e18e1449f90ab942dbace6fcf reported cstate change: term changed from 0 to 1, leader changed from <none> to d572480e18e1449f90ab942dbace6fcf (127.10.131.1). New cstate: current_term: 1 leader_uuid: "d572480e18e1449f90ab942dbace6fcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d572480e18e1449f90ab942dbace6fcf" member_type: VOTER last_known_addr { host: "127.10.131.1" port: 40719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "355a3749b7c441dc9ca5e5ba546bcf40" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5245d05ac0e49f7984122af9b29895e" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 44315 } health_report { overall_health: UNKNOWN } } }
I20260620 18:37:07.974239 31111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:07.968975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33726 (local address 127.10.131.1:40719)
0620 18:37:07.969170 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:07.969175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:07.969213 (+ 38us) server_negotiation.cc:407] Connection header received
0620 18:37:07.969428 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:07.969432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:07.969540 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:07.969681 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:37:07.970000 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:07.970731 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:07.972025 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:07.972467 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:07.972568 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:07.973214 (+ 646us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:37:07.973727 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:37:07.973842 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:07.973976 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 18:37:07.974035 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 18:37:07.981179 31099 tablet.cc:2369] T c21635d0a02f475cac7e71b202f45662 P c5245d05ac0e49f7984122af9b29895e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:37:07.982527 30923 raft_consensus.cc:1261] T c21635d0a02f475cac7e71b202f45662 P 355a3749b7c441dc9ca5e5ba546bcf40 [term 1 FOLLOWER]: Refusing update from remote peer d572480e18e1449f90ab942dbace6fcf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:37:07.982524 31053 raft_consensus.cc:1261] T c21635d0a02f475cac7e71b202f45662 P c5245d05ac0e49f7984122af9b29895e [term 1 FOLLOWER]: Refusing update from remote peer d572480e18e1449f90ab942dbace6fcf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:37:07.983491 31127 consensus_queue.cc:1035] T c21635d0a02f475cac7e71b202f45662 P d572480e18e1449f90ab942dbace6fcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "355a3749b7c441dc9ca5e5ba546bcf40" member_type: VOTER last_known_addr { host: "127.10.131.2" port: 41017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:37:07.983877 31124 consensus_queue.cc:1035] T c21635d0a02f475cac7e71b202f45662 P d572480e18e1449f90ab942dbace6fcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5245d05ac0e49f7984122af9b29895e" member_type: VOTER last_known_addr { host: "127.10.131.3" port: 44315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:37:07.994642 31130 mvcc.cc:205] Tried to move back new op lower bound from 7298992652205727744 to 7298992652050939904. Current Snapshot: MvccSnapshot[applied={T|T < 7298992652205727744}]
I20260620 18:37:08.000651 31132 mvcc.cc:205] Tried to move back new op lower bound from 7298992652205727744 to 7298992652050939904. Current Snapshot: MvccSnapshot[applied={T|T < 7298992652205727744}]
I20260620 18:37:08.007341 30649 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:37:08.007624 30649 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:37:08.011195 30649 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 812daa8a136745809f1c4152f398337f: Sending DeleteTablet for 3 replicas of tablet c21635d0a02f475cac7e71b202f45662
I20260620 18:37:08.012334 30772 tablet_service.cc:1514] Processing DeleteTablet for tablet c21635d0a02f475cac7e71b202f45662 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:37:08 UTC) from {username='slave'} at 127.0.0.1:33720
I20260620 18:37:08.012456 30903 tablet_service.cc:1514] Processing DeleteTablet for tablet c21635d0a02f475cac7e71b202f45662 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:37:08 UTC) from {username='slave'} at 127.0.0.1:43492
I20260620 18:37:08.013269 31140 tablet_replica.cc:330] T c21635d0a02f475cac7e71b202f45662 P d572480e18e1449f90ab942dbace6fcf: stopping tablet replica
I20260620 18:37:08.013540 31033 tablet_service.cc:1514] Processing DeleteTablet for tablet c21635d0a02f475cac7e71b202f45662 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:37:08 UTC) from {username='slave'} at 127.0.0.1:40752
I20260620 18:37:08.013803 31140 raft_consensus.cc:2229] T c21635d0a02f475cac7e71b202f45662 P d572480e18e1449f90ab942dbace6fcf [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:37:08.014453 31140 raft_consensus.cc:2258] T c21635d0a02f475cac7e71b202f45662 P d572480e18e1449f90ab942dbace6fcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:37:08.015616 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 30710
I20260620 18:37:08.016654 31140 ts_tablet_manager.cc:1905] T c21635d0a02f475cac7e71b202f45662 P d572480e18e1449f90ab942dbace6fcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:37:08.022037 31141 tablet_replica.cc:330] T c21635d0a02f475cac7e71b202f45662 P 355a3749b7c441dc9ca5e5ba546bcf40: stopping tablet replica
I20260620 18:37:08.022408 31141 raft_consensus.cc:2229] T c21635d0a02f475cac7e71b202f45662 P 355a3749b7c441dc9ca5e5ba546bcf40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:37:08.022833 31141 raft_consensus.cc:2258] T c21635d0a02f475cac7e71b202f45662 P 355a3749b7c441dc9ca5e5ba546bcf40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:37:08.026093 31142 tablet_replica.cc:330] T c21635d0a02f475cac7e71b202f45662 P c5245d05ac0e49f7984122af9b29895e: stopping tablet replica
I20260620 18:37:08.026505 31142 raft_consensus.cc:2229] T c21635d0a02f475cac7e71b202f45662 P c5245d05ac0e49f7984122af9b29895e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:37:08.026983 31140 ts_tablet_manager.cc:1918] T c21635d0a02f475cac7e71b202f45662 P d572480e18e1449f90ab942dbace6fcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:37:08.027029 31142 raft_consensus.cc:2258] T c21635d0a02f475cac7e71b202f45662 P c5245d05ac0e49f7984122af9b29895e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:37:08.027163 31140 log.cc:1196] T c21635d0a02f475cac7e71b202f45662 P d572480e18e1449f90ab942dbace6fcf: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-0/wal/wals/c21635d0a02f475cac7e71b202f45662
I20260620 18:37:08.028934 31141 ts_tablet_manager.cc:1905] T c21635d0a02f475cac7e71b202f45662 P 355a3749b7c441dc9ca5e5ba546bcf40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:37:08.029093 31142 ts_tablet_manager.cc:1905] T c21635d0a02f475cac7e71b202f45662 P c5245d05ac0e49f7984122af9b29895e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:37:08.034540 31141 ts_tablet_manager.cc:1918] T c21635d0a02f475cac7e71b202f45662 P 355a3749b7c441dc9ca5e5ba546bcf40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:37:08.034634 31142 ts_tablet_manager.cc:1918] T c21635d0a02f475cac7e71b202f45662 P c5245d05ac0e49f7984122af9b29895e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:37:08.034722 31141 log.cc:1196] T c21635d0a02f475cac7e71b202f45662 P 355a3749b7c441dc9ca5e5ba546bcf40: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-1/wal/wals/c21635d0a02f475cac7e71b202f45662
I20260620 18:37:08.034791 31142 log.cc:1196] T c21635d0a02f475cac7e71b202f45662 P c5245d05ac0e49f7984122af9b29895e: Deleting WAL directory at /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781980314095189-10764-0/minicluster-data/ts-2/wal/wals/c21635d0a02f475cac7e71b202f45662
I20260620 18:37:08.035265 31141 ts_tablet_manager.cc:1939] T c21635d0a02f475cac7e71b202f45662 P 355a3749b7c441dc9ca5e5ba546bcf40: Deleting consensus metadata
I20260620 18:37:08.035291 31142 ts_tablet_manager.cc:1939] T c21635d0a02f475cac7e71b202f45662 P c5245d05ac0e49f7984122af9b29895e: Deleting consensus metadata
I20260620 18:37:08.036499 31111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:08.017448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33734 (local address 127.10.131.1:40719)
0620 18:37:08.017666 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:08.017676 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:08.021505 (+ 3829us) server_negotiation.cc:407] Connection header received
0620 18:37:08.021582 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:08.021590 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:08.023184 (+ 1594us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:08.023294 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:08.025567 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:08.026787 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:08.028781 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:08.032583 (+ 3802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:08.032718 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:08.034158 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:08.034168 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:08.034184 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:08.035183 (+ 999us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:08.035244 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:08.035271 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:08.035448 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:08.035998 (+ 550us) server_negotiation.cc:299] Negotiation successful
0620 18:37:08.036108 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 18:37:08.037075 30634 catalog_manager.cc:4841] TS c5245d05ac0e49f7984122af9b29895e (127.10.131.3:44315): tablet c21635d0a02f475cac7e71b202f45662 (table test-table [id=126e0456b2b248feb21bff25856a91fa]) successfully deleted
I20260620 18:37:08.037103 30637 catalog_manager.cc:4841] TS 355a3749b7c441dc9ca5e5ba546bcf40 (127.10.131.2:41017): tablet c21635d0a02f475cac7e71b202f45662 (table test-table [id=126e0456b2b248feb21bff25856a91fa]) successfully deleted
I20260620 18:37:08.037770 30832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:08.130803 31140 ts_tablet_manager.cc:1939] T c21635d0a02f475cac7e71b202f45662 P d572480e18e1449f90ab942dbace6fcf: Deleting consensus metadata
I20260620 18:37:08.131494 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 30710
W20260620 18:37:08.148707 30636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.131.1:40719 (error 108)
I20260620 18:37:08.149398 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 30841
I20260620 18:37:08.150635 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:08.150176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:40719 (local address 127.0.0.1:33746)
0620 18:37:08.150416 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:08.150508 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:40719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260620 18:37:08.150842 30636 catalog_manager.cc:4584] TS d572480e18e1449f90ab942dbace6fcf (127.10.131.1:40719): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c21635d0a02f475cac7e71b202f45662: Network error: Client connection negotiation failed: client connection to 127.10.131.1:40719: connect: Connection refused (error 111)
I20260620 18:37:08.156172 31113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:08.150229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43506 (local address 127.10.131.2:41017)
0620 18:37:08.150413 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:08.150419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:08.150438 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:37:08.150563 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:08.150568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:08.150703 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:08.150801 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:08.151290 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:08.152022 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:08.153419 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:08.153800 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:08.153886 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:08.154852 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:08.154860 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:08.154872 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:08.155442 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:08.155484 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:08.155501 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:08.155592 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:08.155746 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 18:37:08.155826 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:37:08.156982 30963 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:08.204394 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:08.203924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:40719 (local address 127.0.0.1:33750)
0620 18:37:08.204144 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:08.204238 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:40719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 18:37:08.205533 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:08.205134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:40719 (local address 127.0.0.1:33762)
0620 18:37:08.205305 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:08.205418 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:40719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 18:37:08.247210 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 30841
I20260620 18:37:08.264420 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 30971
I20260620 18:37:08.266237 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:08.265708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:40719 (local address 127.0.0.1:33776)
0620 18:37:08.265915 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:08.266017 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:40719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 18:37:08.268353 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:08.267902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:40719 (local address 127.0.0.1:33788)
0620 18:37:08.268095 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:08.268203 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:40719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 18:37:08.272138 31115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:08.265286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40760 (local address 127.10.131.3:44315)
0620 18:37:08.265474 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:08.265482 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:08.265603 (+ 121us) server_negotiation.cc:407] Connection header received
0620 18:37:08.265774 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:08.265779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:08.265900 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:08.266037 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:08.266514 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:08.267263 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:08.269112 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:08.269523 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:08.269664 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:08.270816 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:08.270825 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:08.270840 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:08.271392 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:08.271446 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:08.271467 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:08.271600 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:08.271752 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 18:37:08.271858 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:37:08.272866 31093 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:08.349146 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:08.348567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:40719 (local address 127.0.0.1:33796)
0620 18:37:08.348818 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:08.348961 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:40719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 18:37:08.350256 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:08.349812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.1:40719 (local address 127.0.0.1:33802)
0620 18:37:08.350035 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:08.350119 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.1:40719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260620 18:37:08.361608 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 30971
I20260620 18:37:08.378291 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 30619
I20260620 18:37:08.379407 30680 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:08.487469 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 30619
2026-06-20T18:37:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2720 ms)
[----------] 6 tests from EncryptionPolicyTest (21634 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:37:08 dist-test-slave-cbn3 krb5kdc[31151](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:37:08 dist-test-slave-cbn3 krb5kdc[31151](info): set up 2 sockets
Jun 20 18:37:08 dist-test-slave-cbn3 krb5kdc[31151](info): commencing operation
krb5kdc: starting...
W20260620 18:37:09.593425 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:37:09 dist-test-slave-cbn3 krb5kdc[31151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:37:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:37:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:37:09.856061 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37977
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:10.183059 31164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:10.183442 31164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:10.183609 31164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:10.191556 31164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:37:10.191712 31164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:10.191797 31164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:10.191888 31164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:37:10.191949 31164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:37:10.201824 31164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37977
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:37:10.203186 31164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:10.204489 31164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:10.216284 31169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:10.216312 31170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:10.216393 31172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:10.217099 31164 server_base.cc:1034] running on GCE node
I20260620 18:37:10.217792 31164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:10.219086 31164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:10.220293 31164 hybrid_clock.cc:648] HybridClock initialized: now 1781980630220242 us; error 42 us; skew 500 ppm
I20260620 18:37:10.220592 31164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:10 dist-test-slave-cbn3 krb5kdc[31151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980630, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:10.227907 31164 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:37:10.229672 31164 webserver.cc:460] Webserver started at http://127.0.0.1:46389/ using document root <none> and password file <none>
I20260620 18:37:10.230376 31164 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:10.230513 31164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:10.230851 31164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:10.234004 31164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "5b4728c2b049464689486be5b8f42b56"
format_stamp: "Formatted at 2026-06-20 18:37:10 on dist-test-slave-cbn3"
server_key: "dd173531f778abbb3094194d7e981189"
server_key_iv: "c93b272e3130e73fecdbb90bcde9cd86"
server_key_version: "encryptionkey@0"
I20260620 18:37:10.234817 31164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "5b4728c2b049464689486be5b8f42b56"
format_stamp: "Formatted at 2026-06-20 18:37:10 on dist-test-slave-cbn3"
server_key: "dd173531f778abbb3094194d7e981189"
server_key_iv: "c93b272e3130e73fecdbb90bcde9cd86"
server_key_version: "encryptionkey@0"
I20260620 18:37:10.240013 31164 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:37:10.243777 31179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:10.245141 31164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:37:10.245332 31164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "5b4728c2b049464689486be5b8f42b56"
format_stamp: "Formatted at 2026-06-20 18:37:10 on dist-test-slave-cbn3"
server_key: "dd173531f778abbb3094194d7e981189"
server_key_iv: "c93b272e3130e73fecdbb90bcde9cd86"
server_key_version: "encryptionkey@0"
I20260620 18:37:10.245534 31164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:10.269521 31164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:10.273602 31164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:10.273900 31164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:10.292346 31164 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37977
I20260620 18:37:10.292382 31230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37977 every 8 connection(s)
I20260620 18:37:10.293766 31164 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:37:10.297302 31231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:10.302109 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 31164
I20260620 18:37:10.302363 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:37:10.303215 10764 external_mini_cluster.cc:1428] Setting key f73d1f1bdd5281911abe336754b23ba3
I20260620 18:37:10.308722 31231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b4728c2b049464689486be5b8f42b56: Bootstrap starting.
I20260620 18:37:10.312338 31231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b4728c2b049464689486be5b8f42b56: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:10.313592 31231 log.cc:824] T 00000000000000000000000000000000 P 5b4728c2b049464689486be5b8f42b56: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:10.316464 31231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b4728c2b049464689486be5b8f42b56: No bootstrap required, opened a new log
Jun 20 18:37:10 dist-test-slave-cbn3 krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 18:37:10.327220 31234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:10.305193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41953 (local address 127.0.0.1:37977)
0620 18:37:10.306021 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:10.306035 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:10.306075 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:37:10.307714 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:10.307736 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:10.308133 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:10.308450 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:10.310250 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:10.311516 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:10.313339 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:10.314320 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:10.314520 (+ 200us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:10.318548 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:10.318559 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:10.318578 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:10.318640 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:10.322700 (+ 4060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:10.323475 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:10.323484 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:10.323500 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:10.323705 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:10.324128 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:10.324133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:10.324137 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:10.324833 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:10.325196 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:10.325575 (+ 379us) server_negotiation.cc:299] Negotiation successful
0620 18:37:10.325917 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":413,"threads_started":1}
I20260620 18:37:10.327069 31231 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b4728c2b049464689486be5b8f42b56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4728c2b049464689486be5b8f42b56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37977 } }
I20260620 18:37:10.327534 31231 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b4728c2b049464689486be5b8f42b56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:10.327654 31231 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b4728c2b049464689486be5b8f42b56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b4728c2b049464689486be5b8f42b56, State: Initialized, Role: FOLLOWER
I20260620 18:37:10.328330 31231 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b4728c2b049464689486be5b8f42b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4728c2b049464689486be5b8f42b56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37977 } }
I20260620 18:37:10.328615 31231 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b4728c2b049464689486be5b8f42b56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:37:10.328758 31231 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b4728c2b049464689486be5b8f42b56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:37:10.329007 31231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b4728c2b049464689486be5b8f42b56 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:10.331423 10764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:37:10.331776 31231 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b4728c2b049464689486be5b8f42b56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4728c2b049464689486be5b8f42b56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37977 } }
I20260620 18:37:10.332253 31231 leader_election.cc:302] T 00000000000000000000000000000000 P 5b4728c2b049464689486be5b8f42b56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b4728c2b049464689486be5b8f42b56; no voters:
I20260620 18:37:10.332934 31231 leader_election.cc:288] T 00000000000000000000000000000000 P 5b4728c2b049464689486be5b8f42b56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:37:10.333034 31236 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b4728c2b049464689486be5b8f42b56 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:37:10.334313 31236 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b4728c2b049464689486be5b8f42b56 [term 1 LEADER]: Becoming Leader. State: Replica: 5b4728c2b049464689486be5b8f42b56, State: Running, Role: LEADER
I20260620 18:37:10.334882 31236 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b4728c2b049464689486be5b8f42b56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4728c2b049464689486be5b8f42b56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37977 } }
I20260620 18:37:10.335242 31231 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b4728c2b049464689486be5b8f42b56 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:37:10.337736 31237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b4728c2b049464689486be5b8f42b56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b4728c2b049464689486be5b8f42b56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4728c2b049464689486be5b8f42b56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37977 } } }
I20260620 18:37:10.337705 31238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b4728c2b049464689486be5b8f42b56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b4728c2b049464689486be5b8f42b56. Latest consensus state: current_term: 1 leader_uuid: "5b4728c2b049464689486be5b8f42b56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4728c2b049464689486be5b8f42b56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37977 } } }
I20260620 18:37:10.338136 31238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b4728c2b049464689486be5b8f42b56 [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:10.338136 31237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b4728c2b049464689486be5b8f42b56 [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:10.341351 31247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:37:10.346431 31247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:37:10.363736 31234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:10.337985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59559 (local address 127.0.0.1:37977)
0620 18:37:10.338215 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:10.338224 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:10.341438 (+ 3214us) server_negotiation.cc:407] Connection header received
0620 18:37:10.341633 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:10.341641 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:10.341740 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:10.341851 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:10.344258 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:10.345886 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:10.348319 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:10.349324 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:10.349468 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:10.353836 (+ 4368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:10.353845 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:10.353853 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:10.353903 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:10.358324 (+ 4421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:10.359595 (+ 1271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:10.359604 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:10.359610 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:10.359806 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:10.360483 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:10.360500 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:10.360506 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:10.361012 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:10.361330 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:10.362975 (+ 1645us) server_negotiation.cc:299] Negotiation successful
0620 18:37:10.363208 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:37:10.368191 31247 catalog_manager.cc:1347] Generated new cluster ID: 581890307da84a44a40896a9167bcc4d
I20260620 18:37:10.368301 31247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:37:10.388645 31247 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:37:10.389964 31247 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:37:10.397171 31247 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b4728c2b049464689486be5b8f42b56: Generated new TSK 0
I20260620 18:37:10.398278 31247 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:37:10.412513 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 31164
I20260620 18:37:10.414237 31226 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:10.506098 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 31164
W20260620 18:37:10.523819 31149 connection.cc:535] client connection to 127.0.0.1:37977 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37977): Connection reset by peer (error 104)
2026-06-20T18:37:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2014 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:37:10 dist-test-slave-cbn3 krb5kdc[31263](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:37:10 dist-test-slave-cbn3 krb5kdc[31263](info): set up 2 sockets
Jun 20 18:37:10 dist-test-slave-cbn3 krb5kdc[31263](info): commencing operation
krb5kdc: starting...
W20260620 18:37:11.605439 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.004s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:37:11 dist-test-slave-cbn3 krb5kdc[31263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:37:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:37:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:37:11.855417 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42459
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:12.181939 31276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:12.182286 31276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:12.182471 31276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:12.190044 31276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:37:12.190223 31276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:12.190344 31276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:12.190451 31276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:37:12.190546 31276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:37:12.200807 31276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42459
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:37:12.202297 31276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:12.203656 31276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:12.215602 31281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:12.215553 31284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:12.215651 31282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:12.222427 31276 server_base.cc:1034] running on GCE node
I20260620 18:37:12.223232 31276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:12.224704 31276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:12.225912 31276 hybrid_clock.cc:648] HybridClock initialized: now 1781980632225884 us; error 38 us; skew 500 ppm
I20260620 18:37:12.226274 31276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:12 dist-test-slave-cbn3 krb5kdc[31263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:12.229596 31276 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:37:12.231206 31276 webserver.cc:460] Webserver started at http://127.0.0.1:44179/ using document root <none> and password file <none>
I20260620 18:37:12.231938 31276 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:12.232077 31276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:12.232393 31276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:12.235483 31276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "ea2faa488461423792c161cfc5bd4229"
format_stamp: "Formatted at 2026-06-20 18:37:12 on dist-test-slave-cbn3"
server_key: "a97d01bc0f229a2d2805124f7b8c6bb8"
server_key_iv: "737e4f89f75abeb2e31b6bac879f690d"
server_key_version: "encryptionkey@0"
I20260620 18:37:12.236264 31276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "ea2faa488461423792c161cfc5bd4229"
format_stamp: "Formatted at 2026-06-20 18:37:12 on dist-test-slave-cbn3"
server_key: "a97d01bc0f229a2d2805124f7b8c6bb8"
server_key_iv: "737e4f89f75abeb2e31b6bac879f690d"
server_key_version: "encryptionkey@0"
I20260620 18:37:12.241549 31276 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 18:37:12.244980 31291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:12.246253 31276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:37:12.246448 31276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "ea2faa488461423792c161cfc5bd4229"
format_stamp: "Formatted at 2026-06-20 18:37:12 on dist-test-slave-cbn3"
server_key: "a97d01bc0f229a2d2805124f7b8c6bb8"
server_key_iv: "737e4f89f75abeb2e31b6bac879f690d"
server_key_version: "encryptionkey@0"
I20260620 18:37:12.246650 31276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:12.265365 31276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:12.269999 31276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:12.270301 31276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:12.288800 31276 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42459
I20260620 18:37:12.288823 31342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42459 every 8 connection(s)
I20260620 18:37:12.290228 31276 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:37:12.290819 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 31276
I20260620 18:37:12.291030 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:37:12.291633 10764 external_mini_cluster.cc:1428] Setting key 83572b962508b007022f386551a64192
I20260620 18:37:12.294819 31343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:12.304659 31346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:12.293520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51225 (local address 127.0.0.1:42459)
0620 18:37:12.294300 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:12.294312 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:12.294353 (+ 41us) server_negotiation.cc:407] Connection header received
0620 18:37:12.295995 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:12.296017 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:12.296384 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:12.296766 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:12.297357 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:12.298661 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:12.299998 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:12.300658 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:12.300857 (+ 199us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:12.302368 (+ 1511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:12.302377 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:12.302394 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:12.302954 (+ 560us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:12.303029 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:12.303056 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:12.303259 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:12.303539 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 18:37:12.303709 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":382,"threads_started":1}
I20260620 18:37:12.306171 31343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea2faa488461423792c161cfc5bd4229: Bootstrap starting.
I20260620 18:37:12.308934 10764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:37:12.309825 31343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea2faa488461423792c161cfc5bd4229: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:12.310817 31343 log.cc:824] T 00000000000000000000000000000000 P ea2faa488461423792c161cfc5bd4229: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:12.313985 31343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea2faa488461423792c161cfc5bd4229: No bootstrap required, opened a new log
I20260620 18:37:12.319545 31346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:12.313500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44785 (local address 127.0.0.1:42459)
0620 18:37:12.313692 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:12.313699 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:12.314091 (+ 392us) server_negotiation.cc:407] Connection header received
0620 18:37:12.314316 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:12.314320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:12.314383 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:12.314501 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:12.314983 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:12.315744 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:12.317075 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:12.317497 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:12.317629 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:12.318748 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:12.318753 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:12.318756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:12.318776 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:12.318828 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:12.318846 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:12.318999 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:12.319189 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 18:37:12.319259 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:37:12.322103 31343 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea2faa488461423792c161cfc5bd4229 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea2faa488461423792c161cfc5bd4229" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42459 } }
I20260620 18:37:12.322609 31343 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea2faa488461423792c161cfc5bd4229 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:12.322767 31343 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea2faa488461423792c161cfc5bd4229 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea2faa488461423792c161cfc5bd4229, State: Initialized, Role: FOLLOWER
I20260620 18:37:12.323362 31343 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea2faa488461423792c161cfc5bd4229 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea2faa488461423792c161cfc5bd4229" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42459 } }
I20260620 18:37:12.323597 31343 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea2faa488461423792c161cfc5bd4229 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:37:12.323746 31343 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea2faa488461423792c161cfc5bd4229 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:37:12.323941 31343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea2faa488461423792c161cfc5bd4229 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:12.326179 31343 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea2faa488461423792c161cfc5bd4229 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea2faa488461423792c161cfc5bd4229" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42459 } }
I20260620 18:37:12.326731 31343 leader_election.cc:302] T 00000000000000000000000000000000 P ea2faa488461423792c161cfc5bd4229 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea2faa488461423792c161cfc5bd4229; no voters:
I20260620 18:37:12.327410 31343 leader_election.cc:288] T 00000000000000000000000000000000 P ea2faa488461423792c161cfc5bd4229 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:37:12.327523 31353 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea2faa488461423792c161cfc5bd4229 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:37:12.328243 31353 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea2faa488461423792c161cfc5bd4229 [term 1 LEADER]: Becoming Leader. State: Replica: ea2faa488461423792c161cfc5bd4229, State: Running, Role: LEADER
I20260620 18:37:12.328867 31353 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea2faa488461423792c161cfc5bd4229 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea2faa488461423792c161cfc5bd4229" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42459 } }
I20260620 18:37:12.329708 31343 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea2faa488461423792c161cfc5bd4229 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:37:12.331760 31354 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea2faa488461423792c161cfc5bd4229 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea2faa488461423792c161cfc5bd4229" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea2faa488461423792c161cfc5bd4229" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42459 } } }
I20260620 18:37:12.332130 31354 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea2faa488461423792c161cfc5bd4229 [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:12.334546 31359 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:37:12.334345 31355 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea2faa488461423792c161cfc5bd4229 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea2faa488461423792c161cfc5bd4229. Latest consensus state: current_term: 1 leader_uuid: "ea2faa488461423792c161cfc5bd4229" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea2faa488461423792c161cfc5bd4229" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42459 } } }
I20260620 18:37:12.334785 31355 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea2faa488461423792c161cfc5bd4229 [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:12.340665 31359 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:37:12.350457 31359 catalog_manager.cc:1347] Generated new cluster ID: 5ad6a1ed899542008238156db15512e1
I20260620 18:37:12.350571 31359 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:37:12.357340 31359 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:37:12.359038 31359 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:37:12.368867 31359 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea2faa488461423792c161cfc5bd4229: Generated new TSK 0
I20260620 18:37:12.369683 31359 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:37:12.377887 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 31276
W20260620 18:37:12.378908 31338 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51225
W20260620 18:37:12.379649 10764 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 31276: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:37:12.379845 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 31276
W20260620 18:37:12.396754 31261 connection.cc:535] client connection to 127.0.0.1:42459 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42459): Connection reset by peer (error 104)
2026-06-20T18:37:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1872 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:37:12 dist-test-slave-cbn3 krb5kdc[31374](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:37:12 dist-test-slave-cbn3 krb5kdc[31374](info): set up 2 sockets
Jun 20 18:37:12 dist-test-slave-cbn3 krb5kdc[31374](info): commencing operation
krb5kdc: starting...
W20260620 18:37:13.485198 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:37:13 dist-test-slave-cbn3 krb5kdc[31374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:37:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:37:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:37:13.734264 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42471
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:14.054399 31387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:14.054678 31387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:14.054859 31387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:14.062052 31387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:37:14.062162 31387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:14.062290 31387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:14.062407 31387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:37:14.062505 31387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:37:14.072345 31387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42471
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:37:14.073704 31387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:14.075021 31387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:14.086530 31393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:14.086484 31395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:14.086557 31392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:14.087674 31387 server_base.cc:1034] running on GCE node
I20260620 18:37:14.088449 31387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:14.089874 31387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:14.091121 31387 hybrid_clock.cc:648] HybridClock initialized: now 1781980634091089 us; error 45 us; skew 500 ppm
I20260620 18:37:14.091565 31387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:14 dist-test-slave-cbn3 krb5kdc[31374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980634, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:14.094873 31387 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:37:14.096462 31387 webserver.cc:460] Webserver started at http://127.0.0.1:35263/ using document root <none> and password file <none>
I20260620 18:37:14.097220 31387 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:14.097395 31387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:14.097702 31387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:14.100646 31387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "9bcbc29c573540c69b148677056a0848"
format_stamp: "Formatted at 2026-06-20 18:37:14 on dist-test-slave-cbn3"
server_key: "1dc79453a8190d03fdfd566b2f034067"
server_key_iv: "abe1839d68a3f5e1cf2a1e440462fe74"
server_key_version: "encryptionkey@0"
I20260620 18:37:14.101526 31387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "9bcbc29c573540c69b148677056a0848"
format_stamp: "Formatted at 2026-06-20 18:37:14 on dist-test-slave-cbn3"
server_key: "1dc79453a8190d03fdfd566b2f034067"
server_key_iv: "abe1839d68a3f5e1cf2a1e440462fe74"
server_key_version: "encryptionkey@0"
I20260620 18:37:14.106601 31387 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:37:14.110108 31402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:14.111471 31387 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:37:14.111663 31387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "9bcbc29c573540c69b148677056a0848"
format_stamp: "Formatted at 2026-06-20 18:37:14 on dist-test-slave-cbn3"
server_key: "1dc79453a8190d03fdfd566b2f034067"
server_key_iv: "abe1839d68a3f5e1cf2a1e440462fe74"
server_key_version: "encryptionkey@0"
I20260620 18:37:14.111865 31387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:14.117441 31387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:14.121673 31387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:14.121976 31387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:14.140316 31387 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42471
I20260620 18:37:14.140354 31453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42471 every 8 connection(s)
I20260620 18:37:14.141695 31387 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:37:14.145253 31454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:14.147133 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 31387
I20260620 18:37:14.147341 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:37:14.147766 10764 external_mini_cluster.cc:1428] Setting key 37edbe7982332729d7d77c4105296a4d
I20260620 18:37:14.155690 31457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:14.149636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34299 (local address 127.0.0.1:42471)
0620 18:37:14.150410 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:14.150425 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:14.150475 (+ 50us) server_negotiation.cc:407] Connection header received
0620 18:37:14.152151 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:14.152178 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:14.152600 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:14.153049 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:14.153910 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:14.153918 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:14.153933 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:14.153950 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:14.154022 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:14.154050 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:14.154268 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:14.154382 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 18:37:14.154548 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":334,"threads_started":1}
I20260620 18:37:14.159456 10764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:37:14.159510 31454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bcbc29c573540c69b148677056a0848: Bootstrap starting.
I20260620 18:37:14.162550 31454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bcbc29c573540c69b148677056a0848: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:14.163549 31454 log.cc:824] T 00000000000000000000000000000000 P 9bcbc29c573540c69b148677056a0848: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:14.166417 31457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:14.163778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55645 (local address 127.0.0.1:42471)
0620 18:37:14.163954 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:14.163961 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:14.164447 (+ 486us) server_negotiation.cc:407] Connection header received
0620 18:37:14.164576 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:14.164581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:14.164650 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:14.164784 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:14.165760 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:14.165763 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:14.165767 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:14.165779 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:14.165815 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:14.165829 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:14.165923 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:14.166066 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 18:37:14.166110 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:37:14.166978 31454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bcbc29c573540c69b148677056a0848: No bootstrap required, opened a new log
I20260620 18:37:14.174885 31454 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bcbc29c573540c69b148677056a0848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcbc29c573540c69b148677056a0848" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42471 } }
I20260620 18:37:14.175211 31454 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bcbc29c573540c69b148677056a0848 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:14.175289 31454 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bcbc29c573540c69b148677056a0848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bcbc29c573540c69b148677056a0848, State: Initialized, Role: FOLLOWER
I20260620 18:37:14.175750 31454 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bcbc29c573540c69b148677056a0848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcbc29c573540c69b148677056a0848" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42471 } }
I20260620 18:37:14.175938 31454 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bcbc29c573540c69b148677056a0848 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:37:14.176090 31454 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bcbc29c573540c69b148677056a0848 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:37:14.176223 31454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bcbc29c573540c69b148677056a0848 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:14.178822 31454 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bcbc29c573540c69b148677056a0848 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcbc29c573540c69b148677056a0848" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42471 } }
I20260620 18:37:14.179255 31454 leader_election.cc:302] T 00000000000000000000000000000000 P 9bcbc29c573540c69b148677056a0848 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bcbc29c573540c69b148677056a0848; no voters:
I20260620 18:37:14.179759 31454 leader_election.cc:288] T 00000000000000000000000000000000 P 9bcbc29c573540c69b148677056a0848 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:37:14.179916 31464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bcbc29c573540c69b148677056a0848 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:37:14.180617 31464 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bcbc29c573540c69b148677056a0848 [term 1 LEADER]: Becoming Leader. State: Replica: 9bcbc29c573540c69b148677056a0848, State: Running, Role: LEADER
I20260620 18:37:14.181298 31464 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bcbc29c573540c69b148677056a0848 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcbc29c573540c69b148677056a0848" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42471 } }
I20260620 18:37:14.181954 31454 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bcbc29c573540c69b148677056a0848 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:37:14.184113 31465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bcbc29c573540c69b148677056a0848 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bcbc29c573540c69b148677056a0848" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcbc29c573540c69b148677056a0848" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42471 } } }
I20260620 18:37:14.184487 31465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bcbc29c573540c69b148677056a0848 [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:14.186198 31471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:37:14.186138 31466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bcbc29c573540c69b148677056a0848 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bcbc29c573540c69b148677056a0848. Latest consensus state: current_term: 1 leader_uuid: "9bcbc29c573540c69b148677056a0848" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcbc29c573540c69b148677056a0848" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42471 } } }
I20260620 18:37:14.186455 31466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bcbc29c573540c69b148677056a0848 [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:14.191642 31471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:37:14.201496 31471 catalog_manager.cc:1347] Generated new cluster ID: f24a178fef1f4e428aefe539ef48ee6a
I20260620 18:37:14.201653 31471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:37:14.220587 31471 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:37:14.221086 31471 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:37:14.243614 31471 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bcbc29c573540c69b148677056a0848: Generated new TSK 0
I20260620 18:37:14.244540 31471 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:37:14.248607 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 31387
W20260620 18:37:14.250391 31449 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34299
W20260620 18:37:14.251066 10764 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 31387: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:37:14.251210 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 31387
2026-06-20T18:37:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1870 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:37:14 dist-test-slave-cbn3 krb5kdc[31485](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:37:14 dist-test-slave-cbn3 krb5kdc[31485](info): set up 2 sockets
Jun 20 18:37:14 dist-test-slave-cbn3 krb5kdc[31485](info): commencing operation
krb5kdc: starting...
W20260620 18:37:15.345187 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:37:15 dist-test-slave-cbn3 krb5kdc[31485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:37:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:37:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:37:15.587466 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45755
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:15.912441 31498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:15.912799 31498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:15.912987 31498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:15.920133 31498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:37:15.920287 31498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:15.920403 31498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:15.920516 31498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:37:15.920603 31498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:37:15.930486 31498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:37:15.931766 31498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:15.933077 31498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:15.945043 31503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:15.945014 31506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:15.944980 31504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:15.946053 31498 server_base.cc:1034] running on GCE node
I20260620 18:37:15.946750 31498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:15.948027 31498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:15.949220 31498 hybrid_clock.cc:648] HybridClock initialized: now 1781980635949171 us; error 47 us; skew 500 ppm
I20260620 18:37:15.949558 31498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:15 dist-test-slave-cbn3 krb5kdc[31485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:15.957245 31498 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:37:15.959188 31498 webserver.cc:460] Webserver started at http://127.0.0.1:39423/ using document root <none> and password file <none>
I20260620 18:37:15.960063 31498 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:15.960206 31498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:15.960556 31498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:15.964118 31498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "79c6c0d2a91942eb9a74cdb1fdf97a12"
format_stamp: "Formatted at 2026-06-20 18:37:15 on dist-test-slave-cbn3"
server_key: "4c6c28b4ce5f1629f2a4be1880ee7e00"
server_key_iv: "99e502955eccbd6b7469034fd1999016"
server_key_version: "encryptionkey@0"
I20260620 18:37:15.964968 31498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "79c6c0d2a91942eb9a74cdb1fdf97a12"
format_stamp: "Formatted at 2026-06-20 18:37:15 on dist-test-slave-cbn3"
server_key: "4c6c28b4ce5f1629f2a4be1880ee7e00"
server_key_iv: "99e502955eccbd6b7469034fd1999016"
server_key_version: "encryptionkey@0"
I20260620 18:37:15.970055 31498 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:37:15.973402 31513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:15.974620 31498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 18:37:15.974803 31498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "79c6c0d2a91942eb9a74cdb1fdf97a12"
format_stamp: "Formatted at 2026-06-20 18:37:15 on dist-test-slave-cbn3"
server_key: "4c6c28b4ce5f1629f2a4be1880ee7e00"
server_key_iv: "99e502955eccbd6b7469034fd1999016"
server_key_version: "encryptionkey@0"
I20260620 18:37:15.974999 31498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:15.987591 31498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:15.991425 31498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:15.991691 31498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:16.009800 31498 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45755
I20260620 18:37:16.009891 31564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45755 every 8 connection(s)
I20260620 18:37:16.011178 31498 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:37:16.014840 31565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:16.020177 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 31498
I20260620 18:37:16.020364 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:37:16.020910 10764 external_mini_cluster.cc:1428] Setting key 6646029ee4753c03d88e9432aac4542a
I20260620 18:37:16.026705 31565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79c6c0d2a91942eb9a74cdb1fdf97a12: Bootstrap starting.
I20260620 18:37:16.030113 31565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79c6c0d2a91942eb9a74cdb1fdf97a12: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:16.031127 31565 log.cc:824] T 00000000000000000000000000000000 P 79c6c0d2a91942eb9a74cdb1fdf97a12: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:16.034161 31565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79c6c0d2a91942eb9a74cdb1fdf97a12: No bootstrap required, opened a new log
Jun 20 18:37:16 dist-test-slave-cbn3 krb5kdc[31485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 18:37:16.043258 31565 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79c6c0d2a91942eb9a74cdb1fdf97a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c6c0d2a91942eb9a74cdb1fdf97a12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45755 } }
I20260620 18:37:16.043640 31565 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79c6c0d2a91942eb9a74cdb1fdf97a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:16.043764 31565 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79c6c0d2a91942eb9a74cdb1fdf97a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79c6c0d2a91942eb9a74cdb1fdf97a12, State: Initialized, Role: FOLLOWER
I20260620 18:37:16.043916 31568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:16.022686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47003 (local address 127.0.0.1:45755)
0620 18:37:16.023523 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:16.023533 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:16.023572 (+ 39us) server_negotiation.cc:407] Connection header received
0620 18:37:16.025332 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:16.025355 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:16.025731 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:16.026126 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:16.027951 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:16.029361 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:16.030711 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:16.031442 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:16.031641 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:16.036130 (+ 4489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:16.036137 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:16.036154 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:16.036194 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:16.039594 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:16.040489 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:16.040498 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:16.040505 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:16.040660 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:16.041099 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:16.041103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:16.041105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:16.041662 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:16.041884 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:16.042357 (+ 473us) server_negotiation.cc:299] Negotiation successful
0620 18:37:16.042755 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":386,"threads_started":1}
I20260620 18:37:16.044452 31565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79c6c0d2a91942eb9a74cdb1fdf97a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c6c0d2a91942eb9a74cdb1fdf97a12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45755 } }
I20260620 18:37:16.044741 31565 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79c6c0d2a91942eb9a74cdb1fdf97a12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:37:16.044912 31565 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79c6c0d2a91942eb9a74cdb1fdf97a12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:37:16.045085 31565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79c6c0d2a91942eb9a74cdb1fdf97a12 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:16.047843 10764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:37:16.047863 31565 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79c6c0d2a91942eb9a74cdb1fdf97a12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c6c0d2a91942eb9a74cdb1fdf97a12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45755 } }
I20260620 18:37:16.048388 31565 leader_election.cc:302] T 00000000000000000000000000000000 P 79c6c0d2a91942eb9a74cdb1fdf97a12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79c6c0d2a91942eb9a74cdb1fdf97a12; no voters:
I20260620 18:37:16.049018 31565 leader_election.cc:288] T 00000000000000000000000000000000 P 79c6c0d2a91942eb9a74cdb1fdf97a12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:37:16.049180 31570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79c6c0d2a91942eb9a74cdb1fdf97a12 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:37:16.050915 31565 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79c6c0d2a91942eb9a74cdb1fdf97a12 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:37:16.056145 31570 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79c6c0d2a91942eb9a74cdb1fdf97a12 [term 1 LEADER]: Becoming Leader. State: Replica: 79c6c0d2a91942eb9a74cdb1fdf97a12, State: Running, Role: LEADER
I20260620 18:37:16.056928 31570 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79c6c0d2a91942eb9a74cdb1fdf97a12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c6c0d2a91942eb9a74cdb1fdf97a12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45755 } }
I20260620 18:37:16.060396 31571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79c6c0d2a91942eb9a74cdb1fdf97a12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79c6c0d2a91942eb9a74cdb1fdf97a12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c6c0d2a91942eb9a74cdb1fdf97a12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45755 } } }
I20260620 18:37:16.060886 31571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79c6c0d2a91942eb9a74cdb1fdf97a12 [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:16.061467 31572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79c6c0d2a91942eb9a74cdb1fdf97a12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79c6c0d2a91942eb9a74cdb1fdf97a12. Latest consensus state: current_term: 1 leader_uuid: "79c6c0d2a91942eb9a74cdb1fdf97a12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c6c0d2a91942eb9a74cdb1fdf97a12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45755 } } }
I20260620 18:37:16.061829 31572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79c6c0d2a91942eb9a74cdb1fdf97a12 [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:16.073311 31588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:37:16.084005 31588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:37:16.089461 31568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:16.062739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35035 (local address 127.0.0.1:45755)
0620 18:37:16.062937 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:16.062947 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:16.069603 (+ 6656us) server_negotiation.cc:407] Connection header received
0620 18:37:16.070176 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:16.070184 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:16.070310 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:16.070401 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:16.073083 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:16.074188 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:16.076208 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:16.077189 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:16.077339 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:16.081457 (+ 4118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:16.081463 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:16.081469 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:16.081517 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:16.085565 (+ 4048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:16.086735 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:16.086741 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:16.086744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:16.086899 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:16.087404 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:16.087411 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:16.087419 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:16.087860 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:16.088090 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:16.088705 (+ 615us) server_negotiation.cc:299] Negotiation successful
0620 18:37:16.088952 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 18:37:16.094240 31588 catalog_manager.cc:1347] Generated new cluster ID: c5b0c10bfda14b028a28e91b887eb448
I20260620 18:37:16.094355 31588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:37:16.108738 31588 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:37:16.110087 31588 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:37:16.119351 31588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79c6c0d2a91942eb9a74cdb1fdf97a12: Generated new TSK 0
I20260620 18:37:16.120136 31588 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:37:16.133744 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 31498
I20260620 18:37:16.135105 31560 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:16.226176 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 31498
2026-06-20T18:37:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1975 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:37:16 dist-test-slave-cbn3 krb5kdc[31598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:37:16 dist-test-slave-cbn3 krb5kdc[31598](info): set up 2 sockets
Jun 20 18:37:16 dist-test-slave-cbn3 krb5kdc[31598](info): commencing operation
krb5kdc: starting...
W20260620 18:37:17.337399 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:37:17 dist-test-slave-cbn3 krb5kdc[31598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:37:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:37:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:37:17.586210 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43607
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:17.915109 31611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:17.915465 31611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:17.915639 31611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:17.923040 31611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:37:17.923206 31611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:17.923318 31611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:17.923426 31611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:37:17.923504 31611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:37:17.933696 31611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:37:17.934999 31611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:17.936308 31611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:17.948341 31616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:17.948413 31619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:17.948319 31617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:17.949412 31611 server_base.cc:1034] running on GCE node
I20260620 18:37:17.950205 31611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:17.951581 31611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:17.952819 31611 hybrid_clock.cc:648] HybridClock initialized: now 1781980637952757 us; error 47 us; skew 500 ppm
I20260620 18:37:17.953205 31611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:17 dist-test-slave-cbn3 krb5kdc[31598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:17.956190 31611 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:37:17.957791 31611 webserver.cc:460] Webserver started at http://127.0.0.1:46393/ using document root <none> and password file <none>
I20260620 18:37:17.958434 31611 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:17.958595 31611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:17.958909 31611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:17.961843 31611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "9ab4b9857a3c4776a93268e7c11b1baa"
format_stamp: "Formatted at 2026-06-20 18:37:17 on dist-test-slave-cbn3"
server_key: "f06d316bc84a41c1f3812b0273aefba5"
server_key_iv: "0c70eb143510ed9c928379c288132708"
server_key_version: "encryptionkey@0"
I20260620 18:37:17.962608 31611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "9ab4b9857a3c4776a93268e7c11b1baa"
format_stamp: "Formatted at 2026-06-20 18:37:17 on dist-test-slave-cbn3"
server_key: "f06d316bc84a41c1f3812b0273aefba5"
server_key_iv: "0c70eb143510ed9c928379c288132708"
server_key_version: "encryptionkey@0"
I20260620 18:37:17.967458 31611 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 18:37:17.970755 31626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:17.971992 31611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:37:17.972167 31611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "9ab4b9857a3c4776a93268e7c11b1baa"
format_stamp: "Formatted at 2026-06-20 18:37:17 on dist-test-slave-cbn3"
server_key: "f06d316bc84a41c1f3812b0273aefba5"
server_key_iv: "0c70eb143510ed9c928379c288132708"
server_key_version: "encryptionkey@0"
I20260620 18:37:17.972368 31611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:18.000924 31611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:18.004947 31611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:18.005252 31611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:18.023417 31611 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43607
I20260620 18:37:18.023450 31677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43607 every 8 connection(s)
I20260620 18:37:18.024760 31611 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:37:18.028345 31678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:18.030150 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 31611
I20260620 18:37:18.030365 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:37:18.030800 10764 external_mini_cluster.cc:1428] Setting key da471b41e2606bebd9ab01285984d18f
I20260620 18:37:18.042138 31678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ab4b9857a3c4776a93268e7c11b1baa: Bootstrap starting.
I20260620 18:37:18.043588 31681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:18.032487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37441 (local address 127.0.0.1:43607)
0620 18:37:18.033343 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:18.033355 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:18.033394 (+ 39us) server_negotiation.cc:407] Connection header received
0620 18:37:18.034902 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:18.034922 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:18.035308 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:18.035601 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:18.036155 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:18.037459 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:18.039377 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:18.040000 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:18.040123 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:18.041404 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:18.041411 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:18.041422 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:18.041916 (+ 494us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:18.041984 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:18.042008 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:18.042090 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:18.042430 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 18:37:18.042694 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":425,"threads_started":1}
I20260620 18:37:18.045539 31678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ab4b9857a3c4776a93268e7c11b1baa: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:18.046842 31678 log.cc:824] T 00000000000000000000000000000000 P 9ab4b9857a3c4776a93268e7c11b1baa: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:18.047184 10764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:37:18.049566 31678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ab4b9857a3c4776a93268e7c11b1baa: No bootstrap required, opened a new log
I20260620 18:37:18.058012 31681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:18.051597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35239 (local address 127.0.0.1:43607)
0620 18:37:18.051787 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:18.051794 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:18.052261 (+ 467us) server_negotiation.cc:407] Connection header received
0620 18:37:18.052400 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:18.052405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:18.052488 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:18.052592 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:18.053101 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:18.054190 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:18.055444 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:18.056169 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:18.056296 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:18.057300 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:18.057307 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:18.057312 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:18.057333 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:18.057381 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:18.057406 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:18.057509 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:18.057584 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 18:37:18.057672 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:37:18.059178 31678 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ab4b9857a3c4776a93268e7c11b1baa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab4b9857a3c4776a93268e7c11b1baa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43607 } }
I20260620 18:37:18.059556 31678 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ab4b9857a3c4776a93268e7c11b1baa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:18.059655 31678 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ab4b9857a3c4776a93268e7c11b1baa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ab4b9857a3c4776a93268e7c11b1baa, State: Initialized, Role: FOLLOWER
I20260620 18:37:18.060297 31678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ab4b9857a3c4776a93268e7c11b1baa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab4b9857a3c4776a93268e7c11b1baa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43607 } }
I20260620 18:37:18.060537 31678 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ab4b9857a3c4776a93268e7c11b1baa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:37:18.060708 31678 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ab4b9857a3c4776a93268e7c11b1baa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:37:18.060937 31678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ab4b9857a3c4776a93268e7c11b1baa [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:18.063098 31678 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ab4b9857a3c4776a93268e7c11b1baa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab4b9857a3c4776a93268e7c11b1baa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43607 } }
I20260620 18:37:18.063606 31678 leader_election.cc:302] T 00000000000000000000000000000000 P 9ab4b9857a3c4776a93268e7c11b1baa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ab4b9857a3c4776a93268e7c11b1baa; no voters:
I20260620 18:37:18.064260 31678 leader_election.cc:288] T 00000000000000000000000000000000 P 9ab4b9857a3c4776a93268e7c11b1baa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:37:18.064424 31688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ab4b9857a3c4776a93268e7c11b1baa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:37:18.065551 31688 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ab4b9857a3c4776a93268e7c11b1baa [term 1 LEADER]: Becoming Leader. State: Replica: 9ab4b9857a3c4776a93268e7c11b1baa, State: Running, Role: LEADER
I20260620 18:37:18.066090 31688 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ab4b9857a3c4776a93268e7c11b1baa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab4b9857a3c4776a93268e7c11b1baa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43607 } }
I20260620 18:37:18.066349 31678 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ab4b9857a3c4776a93268e7c11b1baa [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:37:18.069154 31690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ab4b9857a3c4776a93268e7c11b1baa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ab4b9857a3c4776a93268e7c11b1baa. Latest consensus state: current_term: 1 leader_uuid: "9ab4b9857a3c4776a93268e7c11b1baa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab4b9857a3c4776a93268e7c11b1baa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43607 } } }
I20260620 18:37:18.069206 31689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ab4b9857a3c4776a93268e7c11b1baa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ab4b9857a3c4776a93268e7c11b1baa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab4b9857a3c4776a93268e7c11b1baa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43607 } } }
I20260620 18:37:18.069622 31689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ab4b9857a3c4776a93268e7c11b1baa [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:18.069622 31690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ab4b9857a3c4776a93268e7c11b1baa [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:18.072669 31697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:37:18.077708 31697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:37:18.086601 31697 catalog_manager.cc:1347] Generated new cluster ID: abdb7ee987454709851b815fadd5c4d6
I20260620 18:37:18.086714 31697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:37:18.095083 31697 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:37:18.096311 31697 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:37:18.108440 31697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ab4b9857a3c4776a93268e7c11b1baa: Generated new TSK 0
I20260620 18:37:18.109392 31697 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:37:18.120519 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 31611
W20260620 18:37:18.121817 31673 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37441
W20260620 18:37:18.122503 10764 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 31611: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:37:18.122654 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 31611
2026-06-20T18:37:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1895 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:37:18 dist-test-slave-cbn3 krb5kdc[31709](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:37:18 dist-test-slave-cbn3 krb5kdc[31709](info): set up 2 sockets
Jun 20 18:37:18 dist-test-slave-cbn3 krb5kdc[31709](info): commencing operation
krb5kdc: starting...
W20260620 18:37:19.217394 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:37:19 dist-test-slave-cbn3 krb5kdc[31709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:37:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:37:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:37:19.467870 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33391
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:19.795181 31722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:19.795460 31722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:19.795684 31722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:19.803035 31722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:37:19.803175 31722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:19.803292 31722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:19.803411 31722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:37:19.803505 31722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:37:19.813706 31722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:37:19.815018 31722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:19.816341 31722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:19.828688 31727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:19.828692 31730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:19.828734 31728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:19.829200 31722 server_base.cc:1034] running on GCE node
I20260620 18:37:19.830125 31722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:19.831481 31722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:19.832720 31722 hybrid_clock.cc:648] HybridClock initialized: now 1781980639832662 us; error 49 us; skew 500 ppm
I20260620 18:37:19.833105 31722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:19 dist-test-slave-cbn3 krb5kdc[31709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:19.836220 31722 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:37:19.837836 31722 webserver.cc:460] Webserver started at http://127.0.0.1:43633/ using document root <none> and password file <none>
I20260620 18:37:19.838516 31722 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:19.838662 31722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:19.838972 31722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:19.841964 31722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "43f686b66bca42a58d26dcfed62892e8"
format_stamp: "Formatted at 2026-06-20 18:37:19 on dist-test-slave-cbn3"
server_key: "297cf8706c32376344fe955d496c5e5a"
server_key_iv: "878a336f1ddb745940a4549a2dc9e739"
server_key_version: "encryptionkey@0"
I20260620 18:37:19.842743 31722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "43f686b66bca42a58d26dcfed62892e8"
format_stamp: "Formatted at 2026-06-20 18:37:19 on dist-test-slave-cbn3"
server_key: "297cf8706c32376344fe955d496c5e5a"
server_key_iv: "878a336f1ddb745940a4549a2dc9e739"
server_key_version: "encryptionkey@0"
I20260620 18:37:19.847740 31722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:37:19.851075 31737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:19.852289 31722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 18:37:19.852464 31722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "43f686b66bca42a58d26dcfed62892e8"
format_stamp: "Formatted at 2026-06-20 18:37:19 on dist-test-slave-cbn3"
server_key: "297cf8706c32376344fe955d496c5e5a"
server_key_iv: "878a336f1ddb745940a4549a2dc9e739"
server_key_version: "encryptionkey@0"
I20260620 18:37:19.852658 31722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:19.858323 31722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:19.862632 31722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:19.862929 31722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:19.881320 31722 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33391
I20260620 18:37:19.881367 31788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33391 every 8 connection(s)
I20260620 18:37:19.882707 31722 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:37:19.886307 31789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:19.891008 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 31722
I20260620 18:37:19.891196 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:37:19.891737 10764 external_mini_cluster.cc:1428] Setting key 0356d25a46181d496ed4bf7763467470
I20260620 18:37:19.897457 31789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43f686b66bca42a58d26dcfed62892e8: Bootstrap starting.
I20260620 18:37:19.899416 31792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:19.893438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33923 (local address 127.0.0.1:33391)
0620 18:37:19.894250 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:19.894264 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:19.894304 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:37:19.896108 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:19.896141 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:19.896479 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:19.896806 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:19.897894 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:19.897900 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:19.897912 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:19.897926 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:19.897988 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:19.898008 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:19.898171 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:19.898288 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 18:37:19.898480 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":365,"threads_started":1}
I20260620 18:37:19.900743 31789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43f686b66bca42a58d26dcfed62892e8: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:19.901919 31789 log.cc:824] T 00000000000000000000000000000000 P 43f686b66bca42a58d26dcfed62892e8: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:19.903121 10764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:37:19.904641 31789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43f686b66bca42a58d26dcfed62892e8: No bootstrap required, opened a new log
I20260620 18:37:19.909979 31792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:19.907551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44349 (local address 127.0.0.1:33391)
0620 18:37:19.907759 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:19.907766 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:19.908007 (+ 241us) server_negotiation.cc:407] Connection header received
0620 18:37:19.908188 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:19.908191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:19.908240 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:19.908325 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:19.909313 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:19.909317 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:19.909320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:19.909334 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:19.909373 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:19.909387 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:19.909480 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:19.909607 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 18:37:19.909652 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 18:37:19.913677 31789 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43f686b66bca42a58d26dcfed62892e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43f686b66bca42a58d26dcfed62892e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33391 } }
I20260620 18:37:19.914052 31789 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43f686b66bca42a58d26dcfed62892e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:19.914175 31789 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43f686b66bca42a58d26dcfed62892e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43f686b66bca42a58d26dcfed62892e8, State: Initialized, Role: FOLLOWER
I20260620 18:37:19.914789 31789 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43f686b66bca42a58d26dcfed62892e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43f686b66bca42a58d26dcfed62892e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33391 } }
I20260620 18:37:19.915031 31789 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43f686b66bca42a58d26dcfed62892e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:37:19.915186 31789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43f686b66bca42a58d26dcfed62892e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:37:19.915354 31789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43f686b66bca42a58d26dcfed62892e8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:19.918022 31789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43f686b66bca42a58d26dcfed62892e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43f686b66bca42a58d26dcfed62892e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33391 } }
I20260620 18:37:19.918524 31789 leader_election.cc:302] T 00000000000000000000000000000000 P 43f686b66bca42a58d26dcfed62892e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43f686b66bca42a58d26dcfed62892e8; no voters:
I20260620 18:37:19.919077 31789 leader_election.cc:288] T 00000000000000000000000000000000 P 43f686b66bca42a58d26dcfed62892e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:37:19.919241 31799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43f686b66bca42a58d26dcfed62892e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:37:19.919905 31799 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43f686b66bca42a58d26dcfed62892e8 [term 1 LEADER]: Becoming Leader. State: Replica: 43f686b66bca42a58d26dcfed62892e8, State: Running, Role: LEADER
I20260620 18:37:19.920451 31799 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43f686b66bca42a58d26dcfed62892e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43f686b66bca42a58d26dcfed62892e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33391 } }
I20260620 18:37:19.921250 31789 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43f686b66bca42a58d26dcfed62892e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:37:19.923105 31800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43f686b66bca42a58d26dcfed62892e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43f686b66bca42a58d26dcfed62892e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43f686b66bca42a58d26dcfed62892e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33391 } } }
I20260620 18:37:19.923054 31801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43f686b66bca42a58d26dcfed62892e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43f686b66bca42a58d26dcfed62892e8. Latest consensus state: current_term: 1 leader_uuid: "43f686b66bca42a58d26dcfed62892e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43f686b66bca42a58d26dcfed62892e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33391 } } }
I20260620 18:37:19.923563 31800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43f686b66bca42a58d26dcfed62892e8 [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:19.923591 31801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43f686b66bca42a58d26dcfed62892e8 [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:19.924527 31806 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:37:19.930584 31806 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:37:19.938959 31806 catalog_manager.cc:1347] Generated new cluster ID: 9f1dc99927804bf3b4bf494c2154e18e
I20260620 18:37:19.939070 31806 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:37:19.961649 31806 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:37:19.962116 31806 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:37:19.973306 31806 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43f686b66bca42a58d26dcfed62892e8: Generated new TSK 0
I20260620 18:37:19.974321 31806 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:37:19.980501 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 31722
W20260620 18:37:19.981688 31784 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33923
W20260620 18:37:19.982378 10764 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 31722: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:37:19.982519 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 31722
2026-06-20T18:37:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1860 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:37:20 dist-test-slave-cbn3 krb5kdc[31820](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:37:20 dist-test-slave-cbn3 krb5kdc[31820](info): set up 2 sockets
Jun 20 18:37:20 dist-test-slave-cbn3 krb5kdc[31820](info): commencing operation
krb5kdc: starting...
W20260620 18:37:21.093397 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:37:21 dist-test-slave-cbn3 krb5kdc[31820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:37:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:37:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:37:21.340457 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39947
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:21.668164 31833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:21.668498 31833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:21.668682 31833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:21.676000 31833 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:37:21.676157 31833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:21.676272 31833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:21.676388 31833 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:37:21.676487 31833 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:37:21.686443 31833 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:37:21.687775 31833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:21.689103 31833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:21.702875 31838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:21.702850 31839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:21.702950 31841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:21.703804 31833 server_base.cc:1034] running on GCE node
I20260620 18:37:21.704588 31833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:21.706120 31833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:21.707363 31833 hybrid_clock.cc:648] HybridClock initialized: now 1781980641707333 us; error 69 us; skew 500 ppm
I20260620 18:37:21.707726 31833 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:21 dist-test-slave-cbn3 krb5kdc[31820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:21.714517 31833 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:37:21.716101 31833 webserver.cc:460] Webserver started at http://127.0.0.1:34441/ using document root <none> and password file <none>
I20260620 18:37:21.716794 31833 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:21.716982 31833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:21.717353 31833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:21.720378 31833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "0d1cea7ad19a45bcad4ca4266a669e91"
format_stamp: "Formatted at 2026-06-20 18:37:21 on dist-test-slave-cbn3"
server_key: "8b2ec31b7c1c3772f33299abcfabf627"
server_key_iv: "4be859fb72344b1ef106040f261b46e2"
server_key_version: "encryptionkey@0"
I20260620 18:37:21.721230 31833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "0d1cea7ad19a45bcad4ca4266a669e91"
format_stamp: "Formatted at 2026-06-20 18:37:21 on dist-test-slave-cbn3"
server_key: "8b2ec31b7c1c3772f33299abcfabf627"
server_key_iv: "4be859fb72344b1ef106040f261b46e2"
server_key_version: "encryptionkey@0"
I20260620 18:37:21.726114 31833 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 18:37:21.729789 31848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:21.731071 31833 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:37:21.731284 31833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "0d1cea7ad19a45bcad4ca4266a669e91"
format_stamp: "Formatted at 2026-06-20 18:37:21 on dist-test-slave-cbn3"
server_key: "8b2ec31b7c1c3772f33299abcfabf627"
server_key_iv: "4be859fb72344b1ef106040f261b46e2"
server_key_version: "encryptionkey@0"
I20260620 18:37:21.731513 31833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:21.753809 31833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:21.758164 31833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:21.758492 31833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:21.777165 31833 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39947
I20260620 18:37:21.777200 31899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39947 every 8 connection(s)
I20260620 18:37:21.778514 31833 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:37:21.782109 31900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:21.784806 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 31833
I20260620 18:37:21.785027 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:37:21.785458 10764 external_mini_cluster.cc:1428] Setting key a104e93156361d58d918b381e581dc0d
I20260620 18:37:21.792980 31900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d1cea7ad19a45bcad4ca4266a669e91: Bootstrap starting.
I20260620 18:37:21.795751 31900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d1cea7ad19a45bcad4ca4266a669e91: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:21.796773 31900 log.cc:824] T 00000000000000000000000000000000 P 0d1cea7ad19a45bcad4ca4266a669e91: Log is configured to *not* fsync() on all Append() calls
Jun 20 18:37:21 dist-test-slave-cbn3 krb5kdc[31820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 18:37:21.799644 31900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d1cea7ad19a45bcad4ca4266a669e91: No bootstrap required, opened a new log
I20260620 18:37:21.807245 31903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:21.787243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:49663 (local address 127.0.0.1:39947)
0620 18:37:21.787987 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:21.787998 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:21.788036 (+ 38us) server_negotiation.cc:407] Connection header received
0620 18:37:21.789676 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:21.789697 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:21.790073 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:21.790450 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:21.791907 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:21.793298 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:21.794558 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:21.795160 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:21.795356 (+ 196us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:21.799893 (+ 4537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:21.799901 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:21.799914 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:21.799946 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:21.803080 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:21.803921 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:21.803928 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:21.803936 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:21.804099 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:21.804591 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:21.804596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:21.804598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:21.805171 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:21.805517 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:21.806018 (+ 501us) server_negotiation.cc:299] Negotiation successful
0620 18:37:21.806321 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":314,"threads_started":1}
I20260620 18:37:21.808144 31900 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d1cea7ad19a45bcad4ca4266a669e91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d1cea7ad19a45bcad4ca4266a669e91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39947 } }
I20260620 18:37:21.808550 31900 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d1cea7ad19a45bcad4ca4266a669e91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:21.808703 31900 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d1cea7ad19a45bcad4ca4266a669e91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d1cea7ad19a45bcad4ca4266a669e91, State: Initialized, Role: FOLLOWER
I20260620 18:37:21.809274 31900 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d1cea7ad19a45bcad4ca4266a669e91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d1cea7ad19a45bcad4ca4266a669e91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39947 } }
I20260620 18:37:21.809515 31900 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d1cea7ad19a45bcad4ca4266a669e91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:37:21.809693 31900 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d1cea7ad19a45bcad4ca4266a669e91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:37:21.809901 31900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d1cea7ad19a45bcad4ca4266a669e91 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:21.811396 10764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:37:21.812420 31900 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d1cea7ad19a45bcad4ca4266a669e91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d1cea7ad19a45bcad4ca4266a669e91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39947 } }
I20260620 18:37:21.812986 31900 leader_election.cc:302] T 00000000000000000000000000000000 P 0d1cea7ad19a45bcad4ca4266a669e91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d1cea7ad19a45bcad4ca4266a669e91; no voters:
I20260620 18:37:21.813639 31900 leader_election.cc:288] T 00000000000000000000000000000000 P 0d1cea7ad19a45bcad4ca4266a669e91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:37:21.813719 31905 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d1cea7ad19a45bcad4ca4266a669e91 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:37:21.815542 31905 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d1cea7ad19a45bcad4ca4266a669e91 [term 1 LEADER]: Becoming Leader. State: Replica: 0d1cea7ad19a45bcad4ca4266a669e91, State: Running, Role: LEADER
I20260620 18:37:21.816293 31900 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d1cea7ad19a45bcad4ca4266a669e91 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:37:21.816174 31905 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d1cea7ad19a45bcad4ca4266a669e91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d1cea7ad19a45bcad4ca4266a669e91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39947 } }
I20260620 18:37:21.825385 31909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d1cea7ad19a45bcad4ca4266a669e91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d1cea7ad19a45bcad4ca4266a669e91. Latest consensus state: current_term: 1 leader_uuid: "0d1cea7ad19a45bcad4ca4266a669e91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d1cea7ad19a45bcad4ca4266a669e91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39947 } } }
I20260620 18:37:21.825862 31909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d1cea7ad19a45bcad4ca4266a669e91 [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:21.830981 31908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d1cea7ad19a45bcad4ca4266a669e91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d1cea7ad19a45bcad4ca4266a669e91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d1cea7ad19a45bcad4ca4266a669e91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39947 } } }
I20260620 18:37:21.831295 31908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d1cea7ad19a45bcad4ca4266a669e91 [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:21.831446 31920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:37:21.836321 31903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:21.817646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:42053 (local address 127.0.0.1:39947)
0620 18:37:21.817875 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:21.817884 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:21.818384 (+ 500us) server_negotiation.cc:407] Connection header received
0620 18:37:21.818582 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:21.818588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:21.818667 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:21.818773 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:21.820185 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:21.821331 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:21.822902 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:21.823412 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:21.823577 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:21.827545 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:21.827555 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:21.827563 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:21.827616 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:21.832118 (+ 4502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:21.833335 (+ 1217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:21.833342 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:21.833348 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:21.833543 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:21.834149 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:21.834156 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:21.834162 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:21.834667 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:21.835039 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:21.835580 (+ 541us) server_negotiation.cc:299] Negotiation successful
0620 18:37:21.835820 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:37:21.840070 31920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:37:21.850056 31920 catalog_manager.cc:1347] Generated new cluster ID: ffbbbc6af6ea4eefb5d8d66fee6c0fc6
I20260620 18:37:21.850183 31920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:37:21.860991 31920 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:37:21.862501 31920 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:37:21.870728 31920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d1cea7ad19a45bcad4ca4266a669e91: Generated new TSK 0
I20260620 18:37:21.871457 31920 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:37:21.885238 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 31833
I20260620 18:37:21.886682 31895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:21.976047 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 31833
W20260620 18:37:21.992941 31818 connection.cc:535] client connection to 127.0.0.1:39947 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39947): Connection reset by peer (error 104)
2026-06-20T18:37:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1993 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:37:22 dist-test-slave-cbn3 krb5kdc[31933](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:37:22 dist-test-slave-cbn3 krb5kdc[31933](info): set up 2 sockets
Jun 20 18:37:22 dist-test-slave-cbn3 krb5kdc[31933](info): commencing operation
krb5kdc: starting...
W20260620 18:37:23.085196 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:37:23 dist-test-slave-cbn3 krb5kdc[31933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:37:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:37:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:37:23.325991 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33719
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:23.653357 31946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:23.653693 31946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:23.653859 31946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:23.661036 31946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:37:23.661185 31946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:23.661285 31946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:23.661391 31946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:37:23.661477 31946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:37:23.671548 31946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:37:23.672950 31946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:23.674269 31946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:23.686614 31951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:23.686661 31954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:23.686700 31952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:23.687494 31946 server_base.cc:1034] running on GCE node
I20260620 18:37:23.688253 31946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:23.689693 31946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:23.690914 31946 hybrid_clock.cc:648] HybridClock initialized: now 1781980643690875 us; error 51 us; skew 500 ppm
I20260620 18:37:23.691265 31946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:23 dist-test-slave-cbn3 krb5kdc[31933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:23.694280 31946 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:37:23.695709 31946 webserver.cc:460] Webserver started at http://127.0.0.1:36731/ using document root <none> and password file <none>
I20260620 18:37:23.696370 31946 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:23.696511 31946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:23.696797 31946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:23.699719 31946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "849e98edbc2045d19cf2254e6d7b2ae9"
format_stamp: "Formatted at 2026-06-20 18:37:23 on dist-test-slave-cbn3"
server_key: "5f0f324b66d5cc5734fd19b9c7cd185e"
server_key_iv: "d84dfecbf8424e3561494b77e7be1f07"
server_key_version: "encryptionkey@0"
I20260620 18:37:23.700476 31946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "849e98edbc2045d19cf2254e6d7b2ae9"
format_stamp: "Formatted at 2026-06-20 18:37:23 on dist-test-slave-cbn3"
server_key: "5f0f324b66d5cc5734fd19b9c7cd185e"
server_key_iv: "d84dfecbf8424e3561494b77e7be1f07"
server_key_version: "encryptionkey@0"
I20260620 18:37:23.705545 31946 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 18:37:23.708753 31961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:23.709997 31946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:37:23.710186 31946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "849e98edbc2045d19cf2254e6d7b2ae9"
format_stamp: "Formatted at 2026-06-20 18:37:23 on dist-test-slave-cbn3"
server_key: "5f0f324b66d5cc5734fd19b9c7cd185e"
server_key_iv: "d84dfecbf8424e3561494b77e7be1f07"
server_key_version: "encryptionkey@0"
I20260620 18:37:23.710373 31946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:23.737542 31946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:23.741516 31946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:23.741808 31946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:23.759819 31946 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33719
I20260620 18:37:23.759850 32012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33719 every 8 connection(s)
I20260620 18:37:23.761207 31946 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:37:23.764669 32013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:23.769744 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 31946
I20260620 18:37:23.769918 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:37:23.770409 10764 external_mini_cluster.cc:1428] Setting key 752518614cffe67d1ed73393ede73274
I20260620 18:37:23.775453 32013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 849e98edbc2045d19cf2254e6d7b2ae9: Bootstrap starting.
I20260620 18:37:23.778887 32013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 849e98edbc2045d19cf2254e6d7b2ae9: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:23.780014 32013 log.cc:824] T 00000000000000000000000000000000 P 849e98edbc2045d19cf2254e6d7b2ae9: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:23.782965 32013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 849e98edbc2045d19cf2254e6d7b2ae9: No bootstrap required, opened a new log
I20260620 18:37:23.783581 32016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:23.772116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:53467 (local address 127.0.0.1:33719)
0620 18:37:23.772956 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:23.772966 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:23.773013 (+ 47us) server_negotiation.cc:407] Connection header received
0620 18:37:23.774675 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:23.774697 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:23.775074 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:23.775374 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:23.775983 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:23.777414 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:23.779233 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:23.779892 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:23.780013 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:23.781378 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:23.781386 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:23.781399 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:23.781899 (+ 500us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:23.781965 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:23.781992 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:23.782116 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:23.782398 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 18:37:23.782582 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":351,"threads_started":1}
I20260620 18:37:23.787178 10764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:37:23.792284 32013 raft_consensus.cc:348] T 00000000000000000000000000000000 P 849e98edbc2045d19cf2254e6d7b2ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "849e98edbc2045d19cf2254e6d7b2ae9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33719 } }
I20260620 18:37:23.792718 32013 raft_consensus.cc:374] T 00000000000000000000000000000000 P 849e98edbc2045d19cf2254e6d7b2ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:23.792819 32013 raft_consensus.cc:729] T 00000000000000000000000000000000 P 849e98edbc2045d19cf2254e6d7b2ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 849e98edbc2045d19cf2254e6d7b2ae9, State: Initialized, Role: FOLLOWER
I20260620 18:37:23.793459 32013 consensus_queue.cc:260] T 00000000000000000000000000000000 P 849e98edbc2045d19cf2254e6d7b2ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "849e98edbc2045d19cf2254e6d7b2ae9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33719 } }
I20260620 18:37:23.793788 32013 raft_consensus.cc:388] T 00000000000000000000000000000000 P 849e98edbc2045d19cf2254e6d7b2ae9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:37:23.793931 32013 raft_consensus.cc:482] T 00000000000000000000000000000000 P 849e98edbc2045d19cf2254e6d7b2ae9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:37:23.794165 32013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 849e98edbc2045d19cf2254e6d7b2ae9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:23.797243 32013 raft_consensus.cc:504] T 00000000000000000000000000000000 P 849e98edbc2045d19cf2254e6d7b2ae9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "849e98edbc2045d19cf2254e6d7b2ae9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33719 } }
I20260620 18:37:23.797847 32013 leader_election.cc:302] T 00000000000000000000000000000000 P 849e98edbc2045d19cf2254e6d7b2ae9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 849e98edbc2045d19cf2254e6d7b2ae9; no voters:
I20260620 18:37:23.798535 32013 leader_election.cc:288] T 00000000000000000000000000000000 P 849e98edbc2045d19cf2254e6d7b2ae9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:37:23.798640 32023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 849e98edbc2045d19cf2254e6d7b2ae9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:37:23.799489 32016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:23.792365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:60175 (local address 127.0.0.1:33719)
0620 18:37:23.792500 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:23.792504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:23.792974 (+ 470us) server_negotiation.cc:407] Connection header received
0620 18:37:23.793240 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:23.793246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:23.793315 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:23.793389 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:23.794002 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:23.794878 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:23.796706 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:23.797214 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:23.797307 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:23.798591 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:23.798595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:23.798598 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:23.798639 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:23.798706 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:23.798725 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:23.798846 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:23.799113 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 18:37:23.799171 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 18:37:23.801100 32013 sys_catalog.cc:564] T 00000000000000000000000000000000 P 849e98edbc2045d19cf2254e6d7b2ae9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:37:23.805092 32023 raft_consensus.cc:686] T 00000000000000000000000000000000 P 849e98edbc2045d19cf2254e6d7b2ae9 [term 1 LEADER]: Becoming Leader. State: Replica: 849e98edbc2045d19cf2254e6d7b2ae9, State: Running, Role: LEADER
I20260620 18:37:23.805846 32023 consensus_queue.cc:237] T 00000000000000000000000000000000 P 849e98edbc2045d19cf2254e6d7b2ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "849e98edbc2045d19cf2254e6d7b2ae9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33719 } }
I20260620 18:37:23.808889 32024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 849e98edbc2045d19cf2254e6d7b2ae9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "849e98edbc2045d19cf2254e6d7b2ae9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "849e98edbc2045d19cf2254e6d7b2ae9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33719 } } }
I20260620 18:37:23.809267 32024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 849e98edbc2045d19cf2254e6d7b2ae9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:23.809295 32025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 849e98edbc2045d19cf2254e6d7b2ae9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 849e98edbc2045d19cf2254e6d7b2ae9. Latest consensus state: current_term: 1 leader_uuid: "849e98edbc2045d19cf2254e6d7b2ae9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "849e98edbc2045d19cf2254e6d7b2ae9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33719 } } }
I20260620 18:37:23.809625 32025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 849e98edbc2045d19cf2254e6d7b2ae9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:23.811177 32038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:37:23.815431 32037 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 18:37:23.816118 32038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:37:23.824774 32038 catalog_manager.cc:1347] Generated new cluster ID: f13b358d6a014ccc8df47309969ba10a
I20260620 18:37:23.824906 32038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:37:23.834796 32038 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:37:23.836565 32038 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:37:23.847325 32038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 849e98edbc2045d19cf2254e6d7b2ae9: Generated new TSK 0
I20260620 18:37:23.848284 32038 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:37:23.858984 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 31946
W20260620 18:37:23.860131 32008 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.146:53467
W20260620 18:37:23.860872 10764 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 31946: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:37:23.861027 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 31946
W20260620 18:37:23.878504 31931 connection.cc:535] client connection to 127.0.0.1:33719 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33719): Connection reset by peer (error 104)
2026-06-20T18:37:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1885 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:37:23 dist-test-slave-cbn3 krb5kdc[32044](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:37:23 dist-test-slave-cbn3 krb5kdc[32044](info): set up 2 sockets
Jun 20 18:37:23 dist-test-slave-cbn3 krb5kdc[32044](info): commencing operation
krb5kdc: starting...
W20260620 18:37:24.961262 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:37:25 dist-test-slave-cbn3 krb5kdc[32044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:37:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:37:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:37:25.203953 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35587
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:25.524605 32057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:25.525027 32057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:25.525188 32057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:25.533115 32057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:37:25.533232 32057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:25.533313 32057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:25.533401 32057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:37:25.533468 32057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:37:25.543259 32057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:37:25.544472 32057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:25.545751 32057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:25.557605 32063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:25.557710 32065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:25.558295 32062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:25.558426 32057 server_base.cc:1034] running on GCE node
I20260620 18:37:25.559273 32057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:25.560710 32057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:25.561969 32057 hybrid_clock.cc:648] HybridClock initialized: now 1781980645561911 us; error 40 us; skew 500 ppm
I20260620 18:37:25.562348 32057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:25 dist-test-slave-cbn3 krb5kdc[32044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:25.565595 32057 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:37:25.567055 32057 webserver.cc:460] Webserver started at http://127.0.0.1:39233/ using document root <none> and password file <none>
I20260620 18:37:25.567751 32057 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:25.567916 32057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:25.568231 32057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:25.571264 32057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "3a513a610313497080661da7179c5ce3"
format_stamp: "Formatted at 2026-06-20 18:37:25 on dist-test-slave-cbn3"
server_key: "b285d8c3f5ba490120b95c70482bb626"
server_key_iv: "0c6914f6405e7512508f8b09e2b6bbcb"
server_key_version: "encryptionkey@0"
I20260620 18:37:25.572113 32057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "3a513a610313497080661da7179c5ce3"
format_stamp: "Formatted at 2026-06-20 18:37:25 on dist-test-slave-cbn3"
server_key: "b285d8c3f5ba490120b95c70482bb626"
server_key_iv: "0c6914f6405e7512508f8b09e2b6bbcb"
server_key_version: "encryptionkey@0"
I20260620 18:37:25.577847 32057 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260620 18:37:25.581130 32072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:25.582370 32057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:37:25.582547 32057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "3a513a610313497080661da7179c5ce3"
format_stamp: "Formatted at 2026-06-20 18:37:25 on dist-test-slave-cbn3"
server_key: "b285d8c3f5ba490120b95c70482bb626"
server_key_iv: "0c6914f6405e7512508f8b09e2b6bbcb"
server_key_version: "encryptionkey@0"
I20260620 18:37:25.582736 32057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:25.588111 32057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:25.591782 32057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:25.592075 32057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:25.610993 32123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35587 every 8 connection(s)
I20260620 18:37:25.610998 32057 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35587
I20260620 18:37:25.612957 32057 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:37:25.617090 32124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:25.617555 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 32057
I20260620 18:37:25.617733 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:37:25.618259 10764 external_mini_cluster.cc:1428] Setting key 98aff2e9df90632b0a93765a62019c0c
I20260620 18:37:25.625424 32127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:25.619969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:48991 (local address 127.0.0.1:35587)
0620 18:37:25.620721 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:25.620731 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:25.620773 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:37:25.622267 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:25.622286 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:25.622581 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:25.622951 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:25.623905 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:25.623911 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:25.623923 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:25.623936 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:25.623991 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:25.624009 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:25.624259 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:25.624383 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 18:37:25.624554 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":312,"threads_started":1}
I20260620 18:37:25.628763 10764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:37:25.630046 32124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a513a610313497080661da7179c5ce3: Bootstrap starting.
I20260620 18:37:25.633209 32124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a513a610313497080661da7179c5ce3: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:25.634594 32124 log.cc:824] T 00000000000000000000000000000000 P 3a513a610313497080661da7179c5ce3: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:25.635799 32127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:25.633103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:46231 (local address 127.0.0.1:35587)
0620 18:37:25.633283 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:25.633288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:25.633590 (+ 302us) server_negotiation.cc:407] Connection header received
0620 18:37:25.633765 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:25.633769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:25.633812 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:25.633884 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:25.634847 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:25.634851 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:25.634854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:25.634867 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:25.634901 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:25.634915 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:25.635002 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:25.635280 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 18:37:25.635347 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:37:25.637486 32124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a513a610313497080661da7179c5ce3: No bootstrap required, opened a new log
I20260620 18:37:25.644439 32124 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a513a610313497080661da7179c5ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a513a610313497080661da7179c5ce3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35587 } }
I20260620 18:37:25.644797 32124 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a513a610313497080661da7179c5ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:25.644953 32124 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a513a610313497080661da7179c5ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a513a610313497080661da7179c5ce3, State: Initialized, Role: FOLLOWER
I20260620 18:37:25.645505 32124 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a513a610313497080661da7179c5ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a513a610313497080661da7179c5ce3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35587 } }
I20260620 18:37:25.645736 32124 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a513a610313497080661da7179c5ce3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:37:25.645887 32124 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a513a610313497080661da7179c5ce3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:37:25.646066 32124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a513a610313497080661da7179c5ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:25.648164 32124 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a513a610313497080661da7179c5ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a513a610313497080661da7179c5ce3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35587 } }
I20260620 18:37:25.648658 32124 leader_election.cc:302] T 00000000000000000000000000000000 P 3a513a610313497080661da7179c5ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a513a610313497080661da7179c5ce3; no voters:
I20260620 18:37:25.649348 32124 leader_election.cc:288] T 00000000000000000000000000000000 P 3a513a610313497080661da7179c5ce3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:37:25.649443 32134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a513a610313497080661da7179c5ce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:37:25.650094 32134 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a513a610313497080661da7179c5ce3 [term 1 LEADER]: Becoming Leader. State: Replica: 3a513a610313497080661da7179c5ce3, State: Running, Role: LEADER
I20260620 18:37:25.650674 32134 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a513a610313497080661da7179c5ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a513a610313497080661da7179c5ce3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35587 } }
I20260620 18:37:25.651583 32124 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a513a610313497080661da7179c5ce3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:37:25.653450 32135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a513a610313497080661da7179c5ce3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a513a610313497080661da7179c5ce3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a513a610313497080661da7179c5ce3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35587 } } }
I20260620 18:37:25.653818 32135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a513a610313497080661da7179c5ce3 [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:25.657083 32136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a513a610313497080661da7179c5ce3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a513a610313497080661da7179c5ce3. Latest consensus state: current_term: 1 leader_uuid: "3a513a610313497080661da7179c5ce3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a513a610313497080661da7179c5ce3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35587 } } }
I20260620 18:37:25.657526 32136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a513a610313497080661da7179c5ce3 [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:25.658324 32141 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:37:25.663488 32141 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:37:25.673286 32141 catalog_manager.cc:1347] Generated new cluster ID: e65b38cc305b437db83405ccdbc7c7d5
I20260620 18:37:25.673410 32141 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:37:25.706435 32141 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:37:25.707042 32141 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:37:25.719188 32141 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a513a610313497080661da7179c5ce3: Generated new TSK 0
I20260620 18:37:25.720211 32141 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:37:25.738106 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 32057
W20260620 18:37:25.739661 32119 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.146:48991
W20260620 18:37:25.740396 10764 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 32057: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:37:25.740536 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 32057
2026-06-20T18:37:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1878 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:37:25 dist-test-slave-cbn3 krb5kdc[32155](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:37:25 dist-test-slave-cbn3 krb5kdc[32155](info): set up 2 sockets
Jun 20 18:37:25 dist-test-slave-cbn3 krb5kdc[32155](info): commencing operation
krb5kdc: starting...
W20260620 18:37:26.837217 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.008s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:37:26 dist-test-slave-cbn3 krb5kdc[32155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:37:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:37:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:37:27.077538 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37067
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:27.397929 32168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:27.398238 32168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:27.398433 32168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:27.405643 32168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:37:27.405795 32168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:27.405905 32168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:27.406014 32168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:37:27.406090 32168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:37:27.415958 32168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37067
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:37:27.417260 32168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:27.418520 32168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:27.429975 32173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:27.429888 32174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:27.429935 32176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:27.430745 32168 server_base.cc:1034] running on GCE node
I20260620 18:37:27.431435 32168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:27.432793 32168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:27.434031 32168 hybrid_clock.cc:648] HybridClock initialized: now 1781980647433997 us; error 30 us; skew 500 ppm
I20260620 18:37:27.434365 32168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:27 dist-test-slave-cbn3 krb5kdc[32155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:27.440773 32168 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:37:27.442209 32168 webserver.cc:460] Webserver started at http://127.0.0.1:45093/ using document root <none> and password file <none>
I20260620 18:37:27.442873 32168 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:27.443004 32168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:27.443312 32168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:27.446377 32168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "e7a1639cca024416be626dda6adbb859"
format_stamp: "Formatted at 2026-06-20 18:37:27 on dist-test-slave-cbn3"
server_key: "d1f741194dfd43acf4359014851fd11c"
server_key_iv: "7a4ac1f93af54ef0806aaa838c715912"
server_key_version: "encryptionkey@0"
I20260620 18:37:27.447184 32168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "e7a1639cca024416be626dda6adbb859"
format_stamp: "Formatted at 2026-06-20 18:37:27 on dist-test-slave-cbn3"
server_key: "d1f741194dfd43acf4359014851fd11c"
server_key_iv: "7a4ac1f93af54ef0806aaa838c715912"
server_key_version: "encryptionkey@0"
I20260620 18:37:27.452297 32168 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 18:37:27.455730 32183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:27.457067 32168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:37:27.457234 32168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "e7a1639cca024416be626dda6adbb859"
format_stamp: "Formatted at 2026-06-20 18:37:27 on dist-test-slave-cbn3"
server_key: "d1f741194dfd43acf4359014851fd11c"
server_key_iv: "7a4ac1f93af54ef0806aaa838c715912"
server_key_version: "encryptionkey@0"
I20260620 18:37:27.457418 32168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:27.471380 32168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:27.475736 32168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:27.476044 32168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:27.494550 32168 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37067
I20260620 18:37:27.494583 32234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37067 every 8 connection(s)
I20260620 18:37:27.496033 32168 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:37:27.499641 32235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:27.500185 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 32168
I20260620 18:37:27.500365 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:37:27.500933 10764 external_mini_cluster.cc:1428] Setting key fbdd6b3367d76986de1fba3eaf35fb36
I20260620 18:37:27.513589 32235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7a1639cca024416be626dda6adbb859: Bootstrap starting.
Jun 20 18:37:27 dist-test-slave-cbn3 krb5kdc[32155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 18:37:27.517095 32235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7a1639cca024416be626dda6adbb859: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:27.518258 32235 log.cc:824] T 00000000000000000000000000000000 P e7a1639cca024416be626dda6adbb859: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:27.520970 32235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7a1639cca024416be626dda6adbb859: No bootstrap required, opened a new log
I20260620 18:37:27.523932 32238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:27.502648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:43713 (local address 127.0.0.1:37067)
0620 18:37:27.503451 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:27.503462 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:27.503508 (+ 46us) server_negotiation.cc:407] Connection header received
0620 18:37:27.505710 (+ 2202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:27.505732 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:27.506119 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:27.506426 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:27.508206 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:27.509575 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:27.511414 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:27.512060 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:27.512190 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:27.516769 (+ 4579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:27.516776 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:27.516790 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:27.516824 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:27.519857 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:27.520624 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:27.520632 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:27.520637 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:27.520790 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:27.521293 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:27.521305 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:27.521307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:27.521805 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:27.522166 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:27.522676 (+ 510us) server_negotiation.cc:299] Negotiation successful
0620 18:37:27.522968 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":610,"thread_start_us":409,"threads_started":1}
I20260620 18:37:27.527905 10764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:37:27.529553 32235 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7a1639cca024416be626dda6adbb859 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a1639cca024416be626dda6adbb859" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37067 } }
I20260620 18:37:27.529910 32235 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7a1639cca024416be626dda6adbb859 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:27.530018 32235 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7a1639cca024416be626dda6adbb859 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7a1639cca024416be626dda6adbb859, State: Initialized, Role: FOLLOWER
I20260620 18:37:27.530638 32235 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7a1639cca024416be626dda6adbb859 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a1639cca024416be626dda6adbb859" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37067 } }
I20260620 18:37:27.530870 32235 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7a1639cca024416be626dda6adbb859 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:37:27.531013 32235 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7a1639cca024416be626dda6adbb859 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:37:27.531200 32235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7a1639cca024416be626dda6adbb859 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:27.533929 32235 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7a1639cca024416be626dda6adbb859 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a1639cca024416be626dda6adbb859" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37067 } }
I20260620 18:37:27.534461 32235 leader_election.cc:302] T 00000000000000000000000000000000 P e7a1639cca024416be626dda6adbb859 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7a1639cca024416be626dda6adbb859; no voters:
I20260620 18:37:27.535171 32235 leader_election.cc:288] T 00000000000000000000000000000000 P e7a1639cca024416be626dda6adbb859 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:37:27.535288 32245 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7a1639cca024416be626dda6adbb859 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:37:27.536384 32245 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7a1639cca024416be626dda6adbb859 [term 1 LEADER]: Becoming Leader. State: Replica: e7a1639cca024416be626dda6adbb859, State: Running, Role: LEADER
I20260620 18:37:27.537056 32245 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7a1639cca024416be626dda6adbb859 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a1639cca024416be626dda6adbb859" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37067 } }
I20260620 18:37:27.537489 32235 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7a1639cca024416be626dda6adbb859 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:37:27.542569 32247 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7a1639cca024416be626dda6adbb859 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7a1639cca024416be626dda6adbb859. Latest consensus state: current_term: 1 leader_uuid: "e7a1639cca024416be626dda6adbb859" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a1639cca024416be626dda6adbb859" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37067 } } }
I20260620 18:37:27.542990 32247 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7a1639cca024416be626dda6adbb859 [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:27.543686 32246 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7a1639cca024416be626dda6adbb859 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7a1639cca024416be626dda6adbb859" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a1639cca024416be626dda6adbb859" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37067 } } }
I20260620 18:37:27.544098 32246 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7a1639cca024416be626dda6adbb859 [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:27.551403 32238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:27.532351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:50601 (local address 127.0.0.1:37067)
0620 18:37:27.532542 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:27.532548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:27.532967 (+ 419us) server_negotiation.cc:407] Connection header received
0620 18:37:27.533123 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:27.533129 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:27.533203 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:27.533302 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:27.534758 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:27.535870 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:27.537592 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:27.538053 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:27.538187 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:27.541031 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:27.541038 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:27.541043 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:27.541091 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:27.546005 (+ 4914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:27.547280 (+ 1275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:27.547288 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:27.547295 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:27.547470 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:27.548185 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:27.548193 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:27.548199 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:27.548722 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:27.550370 (+ 1648us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:27.550650 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 18:37:27.550899 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:37:27.553398 32252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:37:27.558408 32252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:37:27.568297 32252 catalog_manager.cc:1347] Generated new cluster ID: dd0845f552ea499483c9e2b86fa75607
I20260620 18:37:27.568418 32252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:37:27.577139 32252 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:37:27.578341 32252 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:37:27.586385 32252 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7a1639cca024416be626dda6adbb859: Generated new TSK 0
I20260620 18:37:27.587368 32252 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:37:27.596884 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 32168
I20260620 18:37:27.597999 32230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:27.688014 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 32168
W20260620 18:37:27.704594 32153 connection.cc:535] client connection to 127.0.0.1:37067 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37067): Connection reset by peer (error 104)
2026-06-20T18:37:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1946 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:37:27 dist-test-slave-cbn3 krb5kdc[32267](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:37:27 dist-test-slave-cbn3 krb5kdc[32267](info): set up 2 sockets
Jun 20 18:37:27 dist-test-slave-cbn3 krb5kdc[32267](info): commencing operation
krb5kdc: starting...
W20260620 18:37:28.777629 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:37:28 dist-test-slave-cbn3 krb5kdc[32267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:37:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:37:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:37:29.021397 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40619
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:29.345199 32280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:29.345455 32280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:29.345542 32280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:29.352473 32280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:37:29.352586 32280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:29.352631 32280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:29.352694 32280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:37:29.352736 32280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:37:29.362426 32280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40619
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:37:29.363602 32280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:29.364886 32280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:29.376374 32286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:29.376415 32288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:29.376451 32285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:29.377337 32280 server_base.cc:1034] running on GCE node
I20260620 18:37:29.378146 32280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:29.379467 32280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:29.380681 32280 hybrid_clock.cc:648] HybridClock initialized: now 1781980649380628 us; error 48 us; skew 500 ppm
I20260620 18:37:29.381065 32280 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:29 dist-test-slave-cbn3 krb5kdc[32267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:29.384037 32280 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:37:29.385421 32280 webserver.cc:460] Webserver started at http://127.0.0.1:46139/ using document root <none> and password file <none>
I20260620 18:37:29.386039 32280 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:29.386132 32280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:29.386380 32280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:29.389223 32280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "43a88f6a00e74a1eb4c7943e7b6143af"
format_stamp: "Formatted at 2026-06-20 18:37:29 on dist-test-slave-cbn3"
server_key: "80ddbad1ac342e936d3be38bd7548f1a"
server_key_iv: "dccd579147ca0e313b02e118b07edda0"
server_key_version: "encryptionkey@0"
I20260620 18:37:29.389986 32280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "43a88f6a00e74a1eb4c7943e7b6143af"
format_stamp: "Formatted at 2026-06-20 18:37:29 on dist-test-slave-cbn3"
server_key: "80ddbad1ac342e936d3be38bd7548f1a"
server_key_iv: "dccd579147ca0e313b02e118b07edda0"
server_key_version: "encryptionkey@0"
I20260620 18:37:29.395051 32280 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:37:29.398181 32295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:29.399428 32280 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 18:37:29.399598 32280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "43a88f6a00e74a1eb4c7943e7b6143af"
format_stamp: "Formatted at 2026-06-20 18:37:29 on dist-test-slave-cbn3"
server_key: "80ddbad1ac342e936d3be38bd7548f1a"
server_key_iv: "dccd579147ca0e313b02e118b07edda0"
server_key_version: "encryptionkey@0"
I20260620 18:37:29.399793 32280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:29.434113 32280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:29.438055 32280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:29.438349 32280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:29.456445 32280 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40619
I20260620 18:37:29.456498 32346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40619 every 8 connection(s)
I20260620 18:37:29.457787 32280 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:37:29.461726 32347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:29.465847 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 32280
I20260620 18:37:29.466089 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:37:29.466537 10764 external_mini_cluster.cc:1428] Setting key aaf790fb861e04b94711c9a1fd7ea530
I20260620 18:37:29.474263 32347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43a88f6a00e74a1eb4c7943e7b6143af: Bootstrap starting.
I20260620 18:37:29.478374 32347 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43a88f6a00e74a1eb4c7943e7b6143af: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:29.479761 32347 log.cc:824] T 00000000000000000000000000000000 P 43a88f6a00e74a1eb4c7943e7b6143af: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:29.483402 32347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43a88f6a00e74a1eb4c7943e7b6143af: No bootstrap required, opened a new log
I20260620 18:37:29.483995 32350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:29.468356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:36747 (local address 127.0.0.1:40619)
0620 18:37:29.469149 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:29.469165 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:29.469216 (+ 51us) server_negotiation.cc:407] Connection header received
0620 18:37:29.472352 (+ 3136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:29.472381 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:29.472955 (+ 574us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:29.473371 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:29.474536 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:29.476548 (+ 2012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:29.478597 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:29.479641 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:29.479809 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:29.481814 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:29.481822 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:29.481836 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:29.482344 (+ 508us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:29.482410 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:29.482432 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:29.482601 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:29.482863 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 18:37:29.483044 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":310,"threads_started":1}
I20260620 18:37:29.487974 10764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:37:29.491578 32347 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43a88f6a00e74a1eb4c7943e7b6143af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43a88f6a00e74a1eb4c7943e7b6143af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40619 } }
I20260620 18:37:29.491930 32347 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43a88f6a00e74a1eb4c7943e7b6143af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:29.492048 32347 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43a88f6a00e74a1eb4c7943e7b6143af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43a88f6a00e74a1eb4c7943e7b6143af, State: Initialized, Role: FOLLOWER
I20260620 18:37:29.492671 32347 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43a88f6a00e74a1eb4c7943e7b6143af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43a88f6a00e74a1eb4c7943e7b6143af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40619 } }
I20260620 18:37:29.492940 32347 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43a88f6a00e74a1eb4c7943e7b6143af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:37:29.493078 32347 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43a88f6a00e74a1eb4c7943e7b6143af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:37:29.493254 32347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43a88f6a00e74a1eb4c7943e7b6143af [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:29.495500 32347 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43a88f6a00e74a1eb4c7943e7b6143af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43a88f6a00e74a1eb4c7943e7b6143af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40619 } }
I20260620 18:37:29.496047 32347 leader_election.cc:302] T 00000000000000000000000000000000 P 43a88f6a00e74a1eb4c7943e7b6143af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43a88f6a00e74a1eb4c7943e7b6143af; no voters:
I20260620 18:37:29.496726 32347 leader_election.cc:288] T 00000000000000000000000000000000 P 43a88f6a00e74a1eb4c7943e7b6143af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:37:29.496820 32356 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43a88f6a00e74a1eb4c7943e7b6143af [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:37:29.498307 32356 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43a88f6a00e74a1eb4c7943e7b6143af [term 1 LEADER]: Becoming Leader. State: Replica: 43a88f6a00e74a1eb4c7943e7b6143af, State: Running, Role: LEADER
I20260620 18:37:29.499101 32356 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43a88f6a00e74a1eb4c7943e7b6143af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43a88f6a00e74a1eb4c7943e7b6143af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40619 } }
I20260620 18:37:29.500543 32347 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43a88f6a00e74a1eb4c7943e7b6143af [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:37:29.502133 32359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43a88f6a00e74a1eb4c7943e7b6143af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43a88f6a00e74a1eb4c7943e7b6143af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43a88f6a00e74a1eb4c7943e7b6143af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40619 } } }
I20260620 18:37:29.503684 32359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43a88f6a00e74a1eb4c7943e7b6143af [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:29.502216 32358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43a88f6a00e74a1eb4c7943e7b6143af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43a88f6a00e74a1eb4c7943e7b6143af. Latest consensus state: current_term: 1 leader_uuid: "43a88f6a00e74a1eb4c7943e7b6143af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43a88f6a00e74a1eb4c7943e7b6143af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40619 } } }
I20260620 18:37:29.504215 32358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43a88f6a00e74a1eb4c7943e7b6143af [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:29.504709 32363 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:37:29.508344 32350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:29.497075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:36309 (local address 127.0.0.1:40619)
0620 18:37:29.497315 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:29.497324 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:29.497884 (+ 560us) server_negotiation.cc:407] Connection header received
0620 18:37:29.498147 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:29.498156 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:29.498267 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:29.498385 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:29.499193 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:29.500213 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:29.504464 (+ 4251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:29.505444 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:29.505629 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:29.507172 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:29.507178 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:29.507182 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:29.507206 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:29.507271 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:29.507290 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:29.507409 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:29.507902 (+ 493us) server_negotiation.cc:299] Negotiation successful
0620 18:37:29.507973 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":85}
I20260620 18:37:29.510385 32363 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:37:29.519513 32363 catalog_manager.cc:1347] Generated new cluster ID: e49b40fc628942adafc744bcfb036f31
I20260620 18:37:29.519640 32363 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:37:29.535564 32363 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:37:29.536823 32363 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:37:29.556285 32363 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43a88f6a00e74a1eb4c7943e7b6143af: Generated new TSK 0
I20260620 18:37:29.557205 32363 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:37:29.570009 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 32280
W20260620 18:37:29.571082 32342 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.146:36747
W20260620 18:37:29.571844 10764 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 32280: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:37:29.571995 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 32280
W20260620 18:37:29.588567 32265 connection.cc:535] client connection to 127.0.0.1:40619 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40619): Connection reset by peer (error 104)
2026-06-20T18:37:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1884 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:37:29 dist-test-slave-cbn3 krb5kdc[32378](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:37:29 dist-test-slave-cbn3 krb5kdc[32378](info): set up 2 sockets
Jun 20 18:37:29 dist-test-slave-cbn3 krb5kdc[32378](info): commencing operation
krb5kdc: starting...
W20260620 18:37:30.661347 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:37:30 dist-test-slave-cbn3 krb5kdc[32378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:37:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:37:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:37:30.901361 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40337
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:31.228533 32391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:31.228823 32391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:31.228996 32391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:31.235929 32391 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:37:31.236024 32391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:31.236135 32391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:31.236240 32391 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:37:31.236321 32391 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:37:31.246042 32391 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40337
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980314095189-10764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:37:31.247220 32391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:31.248488 32391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:31.260028 32396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:31.260113 32397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:31.260200 32399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:31.261122 32391 server_base.cc:1034] running on GCE node
I20260620 18:37:31.261888 32391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:31.263362 32391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:31.264568 32391 hybrid_clock.cc:648] HybridClock initialized: now 1781980651264523 us; error 38 us; skew 500 ppm
I20260620 18:37:31.264986 32391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:31 dist-test-slave-cbn3 krb5kdc[32378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:31.268083 32391 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:37:31.269539 32391 webserver.cc:460] Webserver started at http://127.0.0.1:38465/ using document root <none> and password file <none>
I20260620 18:37:31.270193 32391 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:31.270318 32391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:31.270617 32391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:31.273495 32391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "983939e58ae44257a180fd4bc92ec06c"
format_stamp: "Formatted at 2026-06-20 18:37:31 on dist-test-slave-cbn3"
server_key: "fca3da4f37b8b735af792ca61950ef31"
server_key_iv: "813037d1037642f23a7bf5de6399b3ba"
server_key_version: "encryptionkey@0"
I20260620 18:37:31.274292 32391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "983939e58ae44257a180fd4bc92ec06c"
format_stamp: "Formatted at 2026-06-20 18:37:31 on dist-test-slave-cbn3"
server_key: "fca3da4f37b8b735af792ca61950ef31"
server_key_iv: "813037d1037642f23a7bf5de6399b3ba"
server_key_version: "encryptionkey@0"
I20260620 18:37:31.279194 32391 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:37:31.282449 32406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:31.283878 32391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:37:31.284147 32391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "983939e58ae44257a180fd4bc92ec06c"
format_stamp: "Formatted at 2026-06-20 18:37:31 on dist-test-slave-cbn3"
server_key: "fca3da4f37b8b735af792ca61950ef31"
server_key_iv: "813037d1037642f23a7bf5de6399b3ba"
server_key_version: "encryptionkey@0"
I20260620 18:37:31.284452 32391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:31.290043 32391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:31.293825 32391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:31.294066 32391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:31.311918 32391 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40337
I20260620 18:37:31.311967 32457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40337 every 8 connection(s)
I20260620 18:37:31.313349 32391 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:37:31.315129 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 32391
I20260620 18:37:31.315307 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:37:31.315902 10764 external_mini_cluster.cc:1428] Setting key d689f0651d929d1f8553068c337ac51b
I20260620 18:37:31.318051 32458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:31.323340 32461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:31.317682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:39419 (local address 127.0.0.1:40337)
0620 18:37:31.318449 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:31.318460 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:31.318498 (+ 38us) server_negotiation.cc:407] Connection header received
0620 18:37:31.320051 (+ 1553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:31.320070 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:31.320395 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:31.320717 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:31.321795 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:31.321801 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:31.321842 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:31.321861 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:31.321936 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:31.321958 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:31.322137 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:31.322237 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 18:37:31.322450 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":351,"threads_started":1}
I20260620 18:37:31.326838 10764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:37:31.330770 32458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 983939e58ae44257a180fd4bc92ec06c: Bootstrap starting.
I20260620 18:37:31.334422 32458 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 983939e58ae44257a180fd4bc92ec06c: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:31.334470 32461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:31.331942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:47321 (local address 127.0.0.1:40337)
0620 18:37:31.332120 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:31.332126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:31.332593 (+ 467us) server_negotiation.cc:407] Connection header received
0620 18:37:31.332714 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:31.332719 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:31.332780 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:31.332928 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:31.333804 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:31.333809 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:31.333813 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:31.333830 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:31.333875 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:31.333896 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:31.334010 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:31.334094 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 18:37:31.334149 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:37:31.335556 32458 log.cc:824] T 00000000000000000000000000000000 P 983939e58ae44257a180fd4bc92ec06c: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:31.338155 32458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 983939e58ae44257a180fd4bc92ec06c: No bootstrap required, opened a new log
I20260620 18:37:31.345139 32458 raft_consensus.cc:348] T 00000000000000000000000000000000 P 983939e58ae44257a180fd4bc92ec06c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "983939e58ae44257a180fd4bc92ec06c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40337 } }
I20260620 18:37:31.345525 32458 raft_consensus.cc:374] T 00000000000000000000000000000000 P 983939e58ae44257a180fd4bc92ec06c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:31.345634 32458 raft_consensus.cc:729] T 00000000000000000000000000000000 P 983939e58ae44257a180fd4bc92ec06c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 983939e58ae44257a180fd4bc92ec06c, State: Initialized, Role: FOLLOWER
I20260620 18:37:31.346176 32458 consensus_queue.cc:260] T 00000000000000000000000000000000 P 983939e58ae44257a180fd4bc92ec06c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "983939e58ae44257a180fd4bc92ec06c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40337 } }
I20260620 18:37:31.346408 32458 raft_consensus.cc:388] T 00000000000000000000000000000000 P 983939e58ae44257a180fd4bc92ec06c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:37:31.346557 32458 raft_consensus.cc:482] T 00000000000000000000000000000000 P 983939e58ae44257a180fd4bc92ec06c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:37:31.346738 32458 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 983939e58ae44257a180fd4bc92ec06c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:31.348815 32458 raft_consensus.cc:504] T 00000000000000000000000000000000 P 983939e58ae44257a180fd4bc92ec06c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "983939e58ae44257a180fd4bc92ec06c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40337 } }
I20260620 18:37:31.349354 32458 leader_election.cc:302] T 00000000000000000000000000000000 P 983939e58ae44257a180fd4bc92ec06c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 983939e58ae44257a180fd4bc92ec06c; no voters:
I20260620 18:37:31.349910 32458 leader_election.cc:288] T 00000000000000000000000000000000 P 983939e58ae44257a180fd4bc92ec06c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:37:31.350062 32468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 983939e58ae44257a180fd4bc92ec06c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:37:31.350706 32468 raft_consensus.cc:686] T 00000000000000000000000000000000 P 983939e58ae44257a180fd4bc92ec06c [term 1 LEADER]: Becoming Leader. State: Replica: 983939e58ae44257a180fd4bc92ec06c, State: Running, Role: LEADER
I20260620 18:37:31.351208 32468 consensus_queue.cc:237] T 00000000000000000000000000000000 P 983939e58ae44257a180fd4bc92ec06c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "983939e58ae44257a180fd4bc92ec06c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40337 } }
I20260620 18:37:31.351907 32458 sys_catalog.cc:564] T 00000000000000000000000000000000 P 983939e58ae44257a180fd4bc92ec06c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:37:31.354018 32470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 983939e58ae44257a180fd4bc92ec06c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 983939e58ae44257a180fd4bc92ec06c. Latest consensus state: current_term: 1 leader_uuid: "983939e58ae44257a180fd4bc92ec06c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "983939e58ae44257a180fd4bc92ec06c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40337 } } }
I20260620 18:37:31.354212 32469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 983939e58ae44257a180fd4bc92ec06c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "983939e58ae44257a180fd4bc92ec06c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "983939e58ae44257a180fd4bc92ec06c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40337 } } }
I20260620 18:37:31.354504 32470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 983939e58ae44257a180fd4bc92ec06c [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:31.354538 32469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 983939e58ae44257a180fd4bc92ec06c [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:31.358289 32475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:37:31.363410 32475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:37:31.372943 32475 catalog_manager.cc:1347] Generated new cluster ID: d04874d891964c37883e1a5dcf6f80ba
I20260620 18:37:31.373083 32475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:37:31.387734 32475 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:37:31.388187 32475 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:37:31.393901 32475 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 983939e58ae44257a180fd4bc92ec06c: Generated new TSK 0
I20260620 18:37:31.394657 32475 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:37:31.402565 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 32391
W20260620 18:37:31.403574 32453 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.146:39419
W20260620 18:37:31.404241 10764 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 32391: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:37:31.404383 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 32391
2026-06-20T18:37:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1832 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:37:31 dist-test-slave-cbn3 krb5kdc[32489](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:37:31 dist-test-slave-cbn3 krb5kdc[32489](info): set up 2 sockets
Jun 20 18:37:31 dist-test-slave-cbn3 krb5kdc[32489](info): commencing operation
krb5kdc: starting...
W20260620 18:37:32.489686 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:37:32 dist-test-slave-cbn3 krb5kdc[32489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:37:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:37:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.131.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
I20260620 18:37:32.757632 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:34905
--webserver_interface=127.10.131.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:45957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:34905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:33.087035 32502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:33.087404 32502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:33.087591 32502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:33.094872 32502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:37:33.095031 32502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:33.095182 32502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:33.095279 32502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:37:33.095345 32502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:37:33.105337 32502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:45957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:34905
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:34905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:37:33.106686 32502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:33.108080 32502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:33.119728 32507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:33.119817 32508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:33.119760 32510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:33.120683 32502 server_base.cc:1034] running on GCE node
I20260620 18:37:33.121479 32502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:33.122905 32502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:33.124159 32502 hybrid_clock.cc:648] HybridClock initialized: now 1781980653124126 us; error 43 us; skew 500 ppm
I20260620 18:37:33.124547 32502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:33 dist-test-slave-cbn3 krb5kdc[32489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980653, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:33.127775 32502 init.cc:377] Logged in from keytab as kudu/127.10.131.62@KRBTEST.COM (short username kudu)
I20260620 18:37:33.129510 32502 webserver.cc:460] Webserver started at http://127.10.131.62:46293/ using document root <none> and password file <none>
I20260620 18:37:33.130226 32502 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:33.130375 32502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:33.130693 32502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:33.133873 32502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "7b81bfaef8d94203a972c11d46aa1b02"
format_stamp: "Formatted at 2026-06-20 18:37:33 on dist-test-slave-cbn3"
server_key: "92c3ce4d3b7c1b0b21e22302416067f8"
server_key_iv: "f2d81bce119bdd363821571c267fa58c"
server_key_version: "encryptionkey@0"
I20260620 18:37:33.134622 32502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "7b81bfaef8d94203a972c11d46aa1b02"
format_stamp: "Formatted at 2026-06-20 18:37:33 on dist-test-slave-cbn3"
server_key: "92c3ce4d3b7c1b0b21e22302416067f8"
server_key_iv: "f2d81bce119bdd363821571c267fa58c"
server_key_version: "encryptionkey@0"
I20260620 18:37:33.139717 32502 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 18:37:33.143115 32517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:33.144445 32502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:37:33.144645 32502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "7b81bfaef8d94203a972c11d46aa1b02"
format_stamp: "Formatted at 2026-06-20 18:37:33 on dist-test-slave-cbn3"
server_key: "92c3ce4d3b7c1b0b21e22302416067f8"
server_key_iv: "f2d81bce119bdd363821571c267fa58c"
server_key_version: "encryptionkey@0"
I20260620 18:37:33.144891 32502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:33.173689 32502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:33.177542 32502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:33.177836 32502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:33.196413 32502 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:34905
I20260620 18:37:33.196448 32568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:34905 every 8 connection(s)
I20260620 18:37:33.197815 32502 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:37:33.201447 32569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:33.203109 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 32502
I20260620 18:37:33.203285 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:37:33.203823 10764 external_mini_cluster.cc:1428] Setting key b8e9e467115631210bc809286b4a4dd2
I20260620 18:37:33.215843 32569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b81bfaef8d94203a972c11d46aa1b02: Bootstrap starting.
Jun 20 18:37:33 dist-test-slave-cbn3 krb5kdc[32489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:37:33.219229 32569 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b81bfaef8d94203a972c11d46aa1b02: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:33.220377 32569 log.cc:824] T 00000000000000000000000000000000 P 7b81bfaef8d94203a972c11d46aa1b02: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:33.223261 32569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b81bfaef8d94203a972c11d46aa1b02: No bootstrap required, opened a new log
I20260620 18:37:33.227686 32572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:33.205649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50891 (local address 127.10.131.62:34905)
0620 18:37:33.206749 (+ 1100us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:33.206764 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:33.206817 (+ 53us) server_negotiation.cc:407] Connection header received
0620 18:37:33.208488 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:33.208519 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:33.209038 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:33.209472 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:33.210787 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:33.212788 (+ 2001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:33.214054 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:33.215027 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:33.215205 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:33.219693 (+ 4488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:33.219701 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:33.219719 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:33.219764 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:33.222956 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:33.223937 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:33.223944 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:33.223950 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:33.224151 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:33.224696 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:33.224756 (+ 60us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:33.224759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:33.225437 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:33.225695 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:33.226229 (+ 534us) server_negotiation.cc:299] Negotiation successful
0620 18:37:33.226617 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":848,"thread_start_us":466,"threads_started":1}
I20260620 18:37:33.231861 10764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:37:33.232043 32569 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b81bfaef8d94203a972c11d46aa1b02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b81bfaef8d94203a972c11d46aa1b02" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34905 } }
I20260620 18:37:33.232381 32569 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b81bfaef8d94203a972c11d46aa1b02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:33.232489 32569 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b81bfaef8d94203a972c11d46aa1b02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b81bfaef8d94203a972c11d46aa1b02, State: Initialized, Role: FOLLOWER
I20260620 18:37:33.233090 32569 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b81bfaef8d94203a972c11d46aa1b02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b81bfaef8d94203a972c11d46aa1b02" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34905 } }
I20260620 18:37:33.233318 32569 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b81bfaef8d94203a972c11d46aa1b02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:37:33.233466 32569 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b81bfaef8d94203a972c11d46aa1b02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:37:33.233626 32569 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b81bfaef8d94203a972c11d46aa1b02 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:33.236137 32569 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b81bfaef8d94203a972c11d46aa1b02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b81bfaef8d94203a972c11d46aa1b02" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34905 } }
I20260620 18:37:33.236742 32569 leader_election.cc:302] T 00000000000000000000000000000000 P 7b81bfaef8d94203a972c11d46aa1b02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b81bfaef8d94203a972c11d46aa1b02; no voters:
I20260620 18:37:33.237500 32569 leader_election.cc:288] T 00000000000000000000000000000000 P 7b81bfaef8d94203a972c11d46aa1b02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:37:33.237612 32579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b81bfaef8d94203a972c11d46aa1b02 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:37:33.238489 32579 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b81bfaef8d94203a972c11d46aa1b02 [term 1 LEADER]: Becoming Leader. State: Replica: 7b81bfaef8d94203a972c11d46aa1b02, State: Running, Role: LEADER
I20260620 18:37:33.239217 32579 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b81bfaef8d94203a972c11d46aa1b02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b81bfaef8d94203a972c11d46aa1b02" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34905 } }
I20260620 18:37:33.241482 32569 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b81bfaef8d94203a972c11d46aa1b02 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:37:33.243714 32580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b81bfaef8d94203a972c11d46aa1b02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b81bfaef8d94203a972c11d46aa1b02. Latest consensus state: current_term: 1 leader_uuid: "7b81bfaef8d94203a972c11d46aa1b02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b81bfaef8d94203a972c11d46aa1b02" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34905 } } }
I20260620 18:37:33.244557 32580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b81bfaef8d94203a972c11d46aa1b02 [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:33.244269 32581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b81bfaef8d94203a972c11d46aa1b02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b81bfaef8d94203a972c11d46aa1b02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b81bfaef8d94203a972c11d46aa1b02" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34905 } } }
I20260620 18:37:33.245803 32581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b81bfaef8d94203a972c11d46aa1b02 [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:33.246258 32587 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:37:33.251354 32587 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:37:33.257736 32572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:33.236565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39697 (local address 127.10.131.62:34905)
0620 18:37:33.236730 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:33.236734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:33.237317 (+ 583us) server_negotiation.cc:407] Connection header received
0620 18:37:33.237516 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:33.237521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:33.237623 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:33.237726 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:33.239543 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:33.240521 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:33.241913 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:33.242472 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:33.242609 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:33.246579 (+ 3970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:33.246589 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:33.246597 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:33.246649 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:33.251939 (+ 5290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:33.253344 (+ 1405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:33.253353 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:33.253359 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:33.253543 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:33.254247 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:33.254255 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:33.254261 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:33.254834 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:33.255164 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:33.256917 (+ 1753us) server_negotiation.cc:299] Negotiation successful
0620 18:37:33.257175 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":55}
I20260620 18:37:33.264370 32587 catalog_manager.cc:1347] Generated new cluster ID: b57f9ce755bd44cf92c7807592463735
I20260620 18:37:33.264559 32587 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:37:33.281245 32587 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:37:33.283056 32587 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:37:33.294386 32587 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b81bfaef8d94203a972c11d46aa1b02: Generated new TSK 0
I20260620 18:37:33.295364 32587 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:37:33.307569 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 32502
I20260620 18:37:33.308950 32564 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:33.399739 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 32502
W20260620 18:37:33.416589 32487 connection.cc:535] client connection to 127.10.131.62:34905 recv error: Network error: failed to read from TLS socket (remote: 127.10.131.62:34905): Connection reset by peer (error 104)
2026-06-20T18:37:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1995 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:37:33 dist-test-slave-cbn3 krb5kdc[32601](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:37:33 dist-test-slave-cbn3 krb5kdc[32601](info): set up 2 sockets
Jun 20 18:37:33 dist-test-slave-cbn3 krb5kdc[32601](info): commencing operation
krb5kdc: starting...
W20260620 18:37:34.513302 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:37:34 dist-test-slave-cbn3 krb5kdc[32601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:37:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:37:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.131.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
I20260620 18:37:34.757635 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:46529
--webserver_interface=127.10.131.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:33873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:46529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:35.086369 32614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:35.086699 32614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:35.086870 32614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:35.094112 32614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:37:35.094266 32614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:35.094381 32614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:35.094486 32614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:37:35.094563 32614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:37:35.104445 32614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:33873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:46529
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:46529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:37:35.105772 32614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:35.107088 32614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:35.118763 32619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:35.118785 32620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:35.118713 32622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:35.119583 32614 server_base.cc:1034] running on GCE node
I20260620 18:37:35.120383 32614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:35.121773 32614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:35.122970 32614 hybrid_clock.cc:648] HybridClock initialized: now 1781980655122914 us; error 48 us; skew 500 ppm
I20260620 18:37:35.123267 32614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:35 dist-test-slave-cbn3 krb5kdc[32601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980655, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:35.126457 32614 init.cc:377] Logged in from keytab as kudu/127.10.131.62@KRBTEST.COM (short username kudu)
I20260620 18:37:35.127912 32614 webserver.cc:460] Webserver started at http://127.10.131.62:38169/ using document root <none> and password file <none>
I20260620 18:37:35.128597 32614 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:35.128734 32614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:35.129063 32614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:35.132045 32614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "c433ba0f195f486fb48762db22bb9750"
format_stamp: "Formatted at 2026-06-20 18:37:35 on dist-test-slave-cbn3"
server_key: "370b3257bfb392175da2a2d11c760b94"
server_key_iv: "704c41fc039ef632294d520ce1d68e10"
server_key_version: "encryptionkey@0"
I20260620 18:37:35.132825 32614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "c433ba0f195f486fb48762db22bb9750"
format_stamp: "Formatted at 2026-06-20 18:37:35 on dist-test-slave-cbn3"
server_key: "370b3257bfb392175da2a2d11c760b94"
server_key_iv: "704c41fc039ef632294d520ce1d68e10"
server_key_version: "encryptionkey@0"
I20260620 18:37:35.138058 32614 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260620 18:37:35.141371 32629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:35.142750 32614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 18:37:35.142942 32614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "c433ba0f195f486fb48762db22bb9750"
format_stamp: "Formatted at 2026-06-20 18:37:35 on dist-test-slave-cbn3"
server_key: "370b3257bfb392175da2a2d11c760b94"
server_key_iv: "704c41fc039ef632294d520ce1d68e10"
server_key_version: "encryptionkey@0"
I20260620 18:37:35.143141 32614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:35.180377 32614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:35.184536 32614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:35.184855 32614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:35.203213 32614 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:46529
I20260620 18:37:35.203243 32680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:46529 every 8 connection(s)
I20260620 18:37:35.204595 32614 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:37:35.208321 32681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:35.211371 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 32614
I20260620 18:37:35.211584 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:37:35.212044 10764 external_mini_cluster.cc:1428] Setting key 1d21187d9599b83d778888fb365c21be
I20260620 18:37:35.220085 32681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c433ba0f195f486fb48762db22bb9750: Bootstrap starting.
I20260620 18:37:35.223834 32681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c433ba0f195f486fb48762db22bb9750: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:35.225069 32681 log.cc:824] T 00000000000000000000000000000000 P c433ba0f195f486fb48762db22bb9750: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:35.225571 32684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:35.213802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41895 (local address 127.10.131.62:46529)
0620 18:37:35.214538 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:35.214550 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:35.214597 (+ 47us) server_negotiation.cc:407] Connection header received
0620 18:37:35.216254 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:35.216275 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:35.216676 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:35.217053 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:35.217607 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:35.218983 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:35.220926 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:35.221611 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:35.221743 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:35.223096 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:35.223103 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:35.223117 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:35.223634 (+ 517us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:35.223704 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:35.223731 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:35.223851 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:35.224226 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 18:37:35.224439 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":336,"threads_started":1}
I20260620 18:37:35.228096 32681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c433ba0f195f486fb48762db22bb9750: No bootstrap required, opened a new log
I20260620 18:37:35.229451 10764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:37:35.235846 32681 raft_consensus.cc:348] T 00000000000000000000000000000000 P c433ba0f195f486fb48762db22bb9750 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c433ba0f195f486fb48762db22bb9750" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 46529 } }
I20260620 18:37:35.236248 32681 raft_consensus.cc:374] T 00000000000000000000000000000000 P c433ba0f195f486fb48762db22bb9750 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:35.236387 32681 raft_consensus.cc:729] T 00000000000000000000000000000000 P c433ba0f195f486fb48762db22bb9750 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c433ba0f195f486fb48762db22bb9750, State: Initialized, Role: FOLLOWER
I20260620 18:37:35.237078 32681 consensus_queue.cc:260] T 00000000000000000000000000000000 P c433ba0f195f486fb48762db22bb9750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c433ba0f195f486fb48762db22bb9750" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 46529 } }
I20260620 18:37:35.237354 32681 raft_consensus.cc:388] T 00000000000000000000000000000000 P c433ba0f195f486fb48762db22bb9750 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:37:35.237486 32681 raft_consensus.cc:482] T 00000000000000000000000000000000 P c433ba0f195f486fb48762db22bb9750 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:37:35.237630 32681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c433ba0f195f486fb48762db22bb9750 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:35.240633 32684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:35.233807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40577 (local address 127.10.131.62:46529)
0620 18:37:35.233990 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:35.233997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:35.234518 (+ 521us) server_negotiation.cc:407] Connection header received
0620 18:37:35.234737 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:35.234741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:35.234803 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:35.234924 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:35.235395 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:35.236166 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:35.238161 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:35.238606 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:35.238705 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:35.239888 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:35.239893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:35.239896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:35.239914 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:35.239947 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:35.239961 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:35.240041 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:35.240247 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 18:37:35.240287 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 18:37:35.240631 32681 raft_consensus.cc:504] T 00000000000000000000000000000000 P c433ba0f195f486fb48762db22bb9750 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c433ba0f195f486fb48762db22bb9750" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 46529 } }
I20260620 18:37:35.241253 32681 leader_election.cc:302] T 00000000000000000000000000000000 P c433ba0f195f486fb48762db22bb9750 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c433ba0f195f486fb48762db22bb9750; no voters:
I20260620 18:37:35.241904 32681 leader_election.cc:288] T 00000000000000000000000000000000 P c433ba0f195f486fb48762db22bb9750 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:37:35.242025 32691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c433ba0f195f486fb48762db22bb9750 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:37:35.242645 32691 raft_consensus.cc:686] T 00000000000000000000000000000000 P c433ba0f195f486fb48762db22bb9750 [term 1 LEADER]: Becoming Leader. State: Replica: c433ba0f195f486fb48762db22bb9750, State: Running, Role: LEADER
I20260620 18:37:35.243284 32691 consensus_queue.cc:237] T 00000000000000000000000000000000 P c433ba0f195f486fb48762db22bb9750 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c433ba0f195f486fb48762db22bb9750" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 46529 } }
I20260620 18:37:35.244261 32681 sys_catalog.cc:564] T 00000000000000000000000000000000 P c433ba0f195f486fb48762db22bb9750 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:37:35.246284 32692 sys_catalog.cc:455] T 00000000000000000000000000000000 P c433ba0f195f486fb48762db22bb9750 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c433ba0f195f486fb48762db22bb9750" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c433ba0f195f486fb48762db22bb9750" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 46529 } } }
I20260620 18:37:35.246471 32693 sys_catalog.cc:455] T 00000000000000000000000000000000 P c433ba0f195f486fb48762db22bb9750 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c433ba0f195f486fb48762db22bb9750. Latest consensus state: current_term: 1 leader_uuid: "c433ba0f195f486fb48762db22bb9750" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c433ba0f195f486fb48762db22bb9750" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 46529 } } }
I20260620 18:37:35.246757 32692 sys_catalog.cc:458] T 00000000000000000000000000000000 P c433ba0f195f486fb48762db22bb9750 [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:35.246801 32693 sys_catalog.cc:458] T 00000000000000000000000000000000 P c433ba0f195f486fb48762db22bb9750 [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:35.249722 32698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:37:35.256363 32698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:37:35.265708 32698 catalog_manager.cc:1347] Generated new cluster ID: 79c79082424f43cf84f25dfd0782c71d
I20260620 18:37:35.265834 32698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:37:35.288282 32698 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:37:35.289822 32698 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:37:35.302448 32698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c433ba0f195f486fb48762db22bb9750: Generated new TSK 0
I20260620 18:37:35.303208 32698 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:37:35.316264 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 32614
W20260620 18:37:35.317306 32676 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41895
W20260620 18:37:35.318043 10764 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 32614: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:37:35.318192 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 32614
W20260620 18:37:35.335031 32599 connection.cc:535] client connection to 127.10.131.62:46529 recv error: Network error: failed to read from TLS socket (remote: 127.10.131.62:46529): Connection reset by peer (error 104)
2026-06-20T18:37:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1917 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:37:35 dist-test-slave-cbn3 krb5kdc[32712](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:37:35 dist-test-slave-cbn3 krb5kdc[32712](info): set up 2 sockets
Jun 20 18:37:35 dist-test-slave-cbn3 krb5kdc[32712](info): commencing operation
krb5kdc: starting...
W20260620 18:37:36.417330 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.006s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:37:36 dist-test-slave-cbn3 krb5kdc[32712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:37:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:37:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.131.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
I20260620 18:37:36.664757 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:44919
--webserver_interface=127.10.131.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:41209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:44919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:36.991492 32725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:36.991876 32725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:36.992048 32725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:36.999518 32725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:37:36.999706 32725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:36.999823 32725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:36.999934 32725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:37:37.000021 32725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:37:37.010290 32725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:41209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:44919
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:44919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:37:37.011691 32725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:37.013052 32725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:37.024967 32733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:37.025043 32731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:37.025022 32730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:37.025637 32725 server_base.cc:1034] running on GCE node
I20260620 18:37:37.026427 32725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:37.027868 32725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:37.029103 32725 hybrid_clock.cc:648] HybridClock initialized: now 1781980657029046 us; error 46 us; skew 500 ppm
I20260620 18:37:37.029460 32725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:37 dist-test-slave-cbn3 krb5kdc[32712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980657, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:37.032500 32725 init.cc:377] Logged in from keytab as kudu/127.10.131.62@KRBTEST.COM (short username kudu)
I20260620 18:37:37.034089 32725 webserver.cc:460] Webserver started at http://127.10.131.62:40771/ using document root <none> and password file <none>
I20260620 18:37:37.034785 32725 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:37.034934 32725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:37.035246 32725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:37.038365 32725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "b49ad84c4692484e84c793cdbd49d11d"
format_stamp: "Formatted at 2026-06-20 18:37:37 on dist-test-slave-cbn3"
server_key: "d111f5e57fce07ead9a28d91e50f8409"
server_key_iv: "62b017d29239d262663302aa24d8cead"
server_key_version: "encryptionkey@0"
I20260620 18:37:37.039150 32725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "b49ad84c4692484e84c793cdbd49d11d"
format_stamp: "Formatted at 2026-06-20 18:37:37 on dist-test-slave-cbn3"
server_key: "d111f5e57fce07ead9a28d91e50f8409"
server_key_iv: "62b017d29239d262663302aa24d8cead"
server_key_version: "encryptionkey@0"
I20260620 18:37:37.044445 32725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 18:37:37.047885 32740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:37.049245 32725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:37:37.049439 32725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "b49ad84c4692484e84c793cdbd49d11d"
format_stamp: "Formatted at 2026-06-20 18:37:37 on dist-test-slave-cbn3"
server_key: "d111f5e57fce07ead9a28d91e50f8409"
server_key_iv: "62b017d29239d262663302aa24d8cead"
server_key_version: "encryptionkey@0"
I20260620 18:37:37.049640 32725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:37.055490 32725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:37.059484 32725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:37.059782 32725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:37.078487 32725 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:44919
I20260620 18:37:37.078517 323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:44919 every 8 connection(s)
I20260620 18:37:37.080003 32725 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:37:37.083617 324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:37.088240 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 32725
I20260620 18:37:37.088436 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:37:37.089032 10764 external_mini_cluster.cc:1428] Setting key fb3bdfcf55e42dc0f388a7bbcf25ae23
I20260620 18:37:37.094765 324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b49ad84c4692484e84c793cdbd49d11d: Bootstrap starting.
I20260620 18:37:37.096611 327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:37.090669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49745 (local address 127.10.131.62:44919)
0620 18:37:37.091397 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:37.091408 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:37.091449 (+ 41us) server_negotiation.cc:407] Connection header received
0620 18:37:37.093436 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:37.093457 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:37.093770 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:37.094131 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:37.095103 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:37.095109 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:37.095121 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:37.095141 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:37.095203 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:37.095225 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:37.095444 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:37.095569 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 18:37:37.095739 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":318,"threads_started":1}
I20260620 18:37:37.098150 324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b49ad84c4692484e84c793cdbd49d11d: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:37.099299 324 log.cc:824] T 00000000000000000000000000000000 P b49ad84c4692484e84c793cdbd49d11d: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:37.100256 10764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:37:37.102007 324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b49ad84c4692484e84c793cdbd49d11d: No bootstrap required, opened a new log
I20260620 18:37:37.107112 327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:37.104714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53871 (local address 127.10.131.62:44919)
0620 18:37:37.104912 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:37.104918 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:37.105283 (+ 365us) server_negotiation.cc:407] Connection header received
0620 18:37:37.105420 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:37.105425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:37.105480 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:37.105580 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:37.106459 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:37.106470 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:37.106475 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:37.106498 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:37.106547 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:37.106565 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:37.106692 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:37.106765 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 18:37:37.106816 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:37:37.110477 324 raft_consensus.cc:348] T 00000000000000000000000000000000 P b49ad84c4692484e84c793cdbd49d11d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b49ad84c4692484e84c793cdbd49d11d" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44919 } }
I20260620 18:37:37.110807 324 raft_consensus.cc:374] T 00000000000000000000000000000000 P b49ad84c4692484e84c793cdbd49d11d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:37.110880 324 raft_consensus.cc:729] T 00000000000000000000000000000000 P b49ad84c4692484e84c793cdbd49d11d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b49ad84c4692484e84c793cdbd49d11d, State: Initialized, Role: FOLLOWER
I20260620 18:37:37.111353 324 consensus_queue.cc:260] T 00000000000000000000000000000000 P b49ad84c4692484e84c793cdbd49d11d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b49ad84c4692484e84c793cdbd49d11d" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44919 } }
I20260620 18:37:37.111567 324 raft_consensus.cc:388] T 00000000000000000000000000000000 P b49ad84c4692484e84c793cdbd49d11d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:37:37.111660 324 raft_consensus.cc:482] T 00000000000000000000000000000000 P b49ad84c4692484e84c793cdbd49d11d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:37:37.111863 324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b49ad84c4692484e84c793cdbd49d11d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:37.114326 324 raft_consensus.cc:504] T 00000000000000000000000000000000 P b49ad84c4692484e84c793cdbd49d11d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b49ad84c4692484e84c793cdbd49d11d" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44919 } }
I20260620 18:37:37.114845 324 leader_election.cc:302] T 00000000000000000000000000000000 P b49ad84c4692484e84c793cdbd49d11d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b49ad84c4692484e84c793cdbd49d11d; no voters:
I20260620 18:37:37.115394 324 leader_election.cc:288] T 00000000000000000000000000000000 P b49ad84c4692484e84c793cdbd49d11d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:37:37.115533 334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b49ad84c4692484e84c793cdbd49d11d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:37:37.116757 334 raft_consensus.cc:686] T 00000000000000000000000000000000 P b49ad84c4692484e84c793cdbd49d11d [term 1 LEADER]: Becoming Leader. State: Replica: b49ad84c4692484e84c793cdbd49d11d, State: Running, Role: LEADER
I20260620 18:37:37.117256 334 consensus_queue.cc:237] T 00000000000000000000000000000000 P b49ad84c4692484e84c793cdbd49d11d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b49ad84c4692484e84c793cdbd49d11d" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44919 } }
I20260620 18:37:37.117602 324 sys_catalog.cc:564] T 00000000000000000000000000000000 P b49ad84c4692484e84c793cdbd49d11d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:37:37.120481 336 sys_catalog.cc:455] T 00000000000000000000000000000000 P b49ad84c4692484e84c793cdbd49d11d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b49ad84c4692484e84c793cdbd49d11d. Latest consensus state: current_term: 1 leader_uuid: "b49ad84c4692484e84c793cdbd49d11d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b49ad84c4692484e84c793cdbd49d11d" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44919 } } }
I20260620 18:37:37.120448 335 sys_catalog.cc:455] T 00000000000000000000000000000000 P b49ad84c4692484e84c793cdbd49d11d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b49ad84c4692484e84c793cdbd49d11d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b49ad84c4692484e84c793cdbd49d11d" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 44919 } } }
I20260620 18:37:37.120921 336 sys_catalog.cc:458] T 00000000000000000000000000000000 P b49ad84c4692484e84c793cdbd49d11d [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:37.120944 335 sys_catalog.cc:458] T 00000000000000000000000000000000 P b49ad84c4692484e84c793cdbd49d11d [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:37.122854 342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:37:37.128245 342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:37:37.137205 342 catalog_manager.cc:1347] Generated new cluster ID: 0997ec34363047c7b9b2182ca59a4487
I20260620 18:37:37.137302 342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:37:37.161265 342 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:37:37.161724 342 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:37:37.176750 342 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b49ad84c4692484e84c793cdbd49d11d: Generated new TSK 0
I20260620 18:37:37.177713 342 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:37:37.186794 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 32725
W20260620 18:37:37.187791 319 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49745
W20260620 18:37:37.188442 10764 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 32725: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:37:37.188596 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 32725
2026-06-20T18:37:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1870 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:37:37 dist-test-slave-cbn3 krb5kdc[355](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:37:37 dist-test-slave-cbn3 krb5kdc[355](info): set up 2 sockets
Jun 20 18:37:37 dist-test-slave-cbn3 krb5kdc[355](info): commencing operation
krb5kdc: starting...
W20260620 18:37:38.289413 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:37:38 dist-test-slave-cbn3 krb5kdc[355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:37:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:37:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.131.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
I20260620 18:37:38.540179 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:43585
--webserver_interface=127.10.131.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:38231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:43585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:38.868562 368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:38.868992 368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:38.869197 368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:38.876937 368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:37:38.877068 368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:38.877189 368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:38.877302 368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:37:38.877400 368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:37:38.888094 368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:38231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:43585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:43585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:37:38.889622 368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:38.891001 368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:38.903280 373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:38.903235 376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:38.903959 374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:38.904326 368 server_base.cc:1034] running on GCE node
I20260620 18:37:38.905095 368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:38.906410 368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:38.907609 368 hybrid_clock.cc:648] HybridClock initialized: now 1781980658907535 us; error 66 us; skew 500 ppm
I20260620 18:37:38.907984 368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:38 dist-test-slave-cbn3 krb5kdc[355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980658, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:38.915457 368 init.cc:377] Logged in from keytab as kudu/127.10.131.62@KRBTEST.COM (short username kudu)
I20260620 18:37:38.917008 368 webserver.cc:460] Webserver started at http://127.10.131.62:38271/ using document root <none> and password file <none>
I20260620 18:37:38.917724 368 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:38.917881 368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:38.918278 368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:38.921412 368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "11f81d041c3a428c97c2203fa4cd61da"
format_stamp: "Formatted at 2026-06-20 18:37:38 on dist-test-slave-cbn3"
server_key: "c646bb21ab55da424a7cdfca57b31b6d"
server_key_iv: "970cf49db2a2201970fd40890bc00ebd"
server_key_version: "encryptionkey@0"
I20260620 18:37:38.922168 368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "11f81d041c3a428c97c2203fa4cd61da"
format_stamp: "Formatted at 2026-06-20 18:37:38 on dist-test-slave-cbn3"
server_key: "c646bb21ab55da424a7cdfca57b31b6d"
server_key_iv: "970cf49db2a2201970fd40890bc00ebd"
server_key_version: "encryptionkey@0"
I20260620 18:37:38.927392 368 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:37:38.930747 383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:38.932068 368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:37:38.932268 368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "11f81d041c3a428c97c2203fa4cd61da"
format_stamp: "Formatted at 2026-06-20 18:37:38 on dist-test-slave-cbn3"
server_key: "c646bb21ab55da424a7cdfca57b31b6d"
server_key_iv: "970cf49db2a2201970fd40890bc00ebd"
server_key_version: "encryptionkey@0"
I20260620 18:37:38.932466 368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:38.954686 368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:38.959339 368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:38.959645 368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:38.978479 368 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:43585
I20260620 18:37:38.978500 434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:43585 every 8 connection(s)
I20260620 18:37:38.979864 368 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:37:38.983526 435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:38.985894 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 368
I20260620 18:37:38.986084 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:37:38.986538 10764 external_mini_cluster.cc:1428] Setting key ec6c910b817ff0686056f5e07d993147
I20260620 18:37:38.994664 435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11f81d041c3a428c97c2203fa4cd61da: Bootstrap starting.
I20260620 18:37:38.997745 435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11f81d041c3a428c97c2203fa4cd61da: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:38.998718 435 log.cc:824] T 00000000000000000000000000000000 P 11f81d041c3a428c97c2203fa4cd61da: Log is configured to *not* fsync() on all Append() calls
Jun 20 18:37:38 dist-test-slave-cbn3 krb5kdc[355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:37:39.001829 435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11f81d041c3a428c97c2203fa4cd61da: No bootstrap required, opened a new log
I20260620 18:37:39.008649 438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:38.988410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38441 (local address 127.10.131.62:43585)
0620 18:37:38.989256 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:38.989270 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:38.989310 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:37:38.990977 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:38.990999 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:38.991400 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:38.991790 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:38.993312 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:38.994623 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:38.995879 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:38.996561 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:38.996752 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:39.001265 (+ 4513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:39.001272 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:39.001287 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:39.001325 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:39.004507 (+ 3182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:39.005418 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:39.005427 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:39.005433 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:39.005610 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:39.006008 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:39.006011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:39.006013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:39.006544 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:39.006778 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:39.007208 (+ 430us) server_negotiation.cc:299] Negotiation successful
0620 18:37:39.007628 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":410,"threads_started":1}
I20260620 18:37:39.010517 435 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11f81d041c3a428c97c2203fa4cd61da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11f81d041c3a428c97c2203fa4cd61da" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 43585 } }
I20260620 18:37:39.010887 435 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11f81d041c3a428c97c2203fa4cd61da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:39.011024 435 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11f81d041c3a428c97c2203fa4cd61da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11f81d041c3a428c97c2203fa4cd61da, State: Initialized, Role: FOLLOWER
I20260620 18:37:39.011679 435 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11f81d041c3a428c97c2203fa4cd61da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11f81d041c3a428c97c2203fa4cd61da" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 43585 } }
I20260620 18:37:39.011933 435 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11f81d041c3a428c97c2203fa4cd61da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:37:39.012099 435 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11f81d041c3a428c97c2203fa4cd61da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:37:39.012256 10764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:37:39.012302 435 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11f81d041c3a428c97c2203fa4cd61da [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:39.014514 435 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11f81d041c3a428c97c2203fa4cd61da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11f81d041c3a428c97c2203fa4cd61da" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 43585 } }
I20260620 18:37:39.015105 435 leader_election.cc:302] T 00000000000000000000000000000000 P 11f81d041c3a428c97c2203fa4cd61da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11f81d041c3a428c97c2203fa4cd61da; no voters:
I20260620 18:37:39.015761 435 leader_election.cc:288] T 00000000000000000000000000000000 P 11f81d041c3a428c97c2203fa4cd61da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:37:39.015856 444 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11f81d041c3a428c97c2203fa4cd61da [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:37:39.017326 444 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11f81d041c3a428c97c2203fa4cd61da [term 1 LEADER]: Becoming Leader. State: Replica: 11f81d041c3a428c97c2203fa4cd61da, State: Running, Role: LEADER
I20260620 18:37:39.017979 444 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11f81d041c3a428c97c2203fa4cd61da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11f81d041c3a428c97c2203fa4cd61da" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 43585 } }
I20260620 18:37:39.018201 435 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11f81d041c3a428c97c2203fa4cd61da [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:37:39.021750 445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11f81d041c3a428c97c2203fa4cd61da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11f81d041c3a428c97c2203fa4cd61da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11f81d041c3a428c97c2203fa4cd61da" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 43585 } } }
I20260620 18:37:39.022207 445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11f81d041c3a428c97c2203fa4cd61da [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:39.022804 446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11f81d041c3a428c97c2203fa4cd61da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11f81d041c3a428c97c2203fa4cd61da. Latest consensus state: current_term: 1 leader_uuid: "11f81d041c3a428c97c2203fa4cd61da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11f81d041c3a428c97c2203fa4cd61da" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 43585 } } }
I20260620 18:37:39.023128 446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11f81d041c3a428c97c2203fa4cd61da [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:39.033049 453 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:37:39.036585 438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:39.017096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35135 (local address 127.10.131.62:43585)
0620 18:37:39.017271 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:39.017276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:39.018096 (+ 820us) server_negotiation.cc:407] Connection header received
0620 18:37:39.018336 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:39.018341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:39.018409 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:39.018462 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:39.020190 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:39.021393 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:39.023190 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:39.024145 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:39.024267 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:39.028597 (+ 4330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:39.028607 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:39.028615 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:39.028666 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:39.033178 (+ 4512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:39.033987 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:39.033992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:39.033995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:39.034136 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:39.034583 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:39.034587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:39.034590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:39.035021 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:39.035314 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:39.035956 (+ 642us) server_negotiation.cc:299] Negotiation successful
0620 18:37:39.036185 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 18:37:39.038902 453 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:37:39.047369 453 catalog_manager.cc:1347] Generated new cluster ID: 5621ca0a96fe4e7ca9917aa5e20c3338
I20260620 18:37:39.047508 453 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:37:39.068781 453 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:37:39.070129 453 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:37:39.079957 453 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11f81d041c3a428c97c2203fa4cd61da: Generated new TSK 0
I20260620 18:37:39.080785 453 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:37:39.087536 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 368
I20260620 18:37:39.088645 430 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:39.179217 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 368
2026-06-20T18:37:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1988 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:37:39 dist-test-slave-cbn3 krb5kdc[467](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:37:39 dist-test-slave-cbn3 krb5kdc[467](info): set up 2 sockets
Jun 20 18:37:39 dist-test-slave-cbn3 krb5kdc[467](info): commencing operation
krb5kdc: starting...
W20260620 18:37:40.277422 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:37:40 dist-test-slave-cbn3 krb5kdc[467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:37:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:37:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.131.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
I20260620 18:37:40.540000 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:45033
--webserver_interface=127.10.131.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:45219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:45033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:40.875708 480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:40.876024 480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:40.876120 480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:40.883524 480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:37:40.883656 480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:40.883710 480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:40.883829 480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:37:40.883896 480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:37:40.894316 480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:45219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:45033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:45033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:37:40.895614 480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:40.896924 480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:40.909098 485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:40.909041 488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:40.909068 486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:40.910154 480 server_base.cc:1034] running on GCE node
I20260620 18:37:40.911049 480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:40.912561 480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:40.913825 480 hybrid_clock.cc:648] HybridClock initialized: now 1781980660913722 us; error 62 us; skew 500 ppm
I20260620 18:37:40.914192 480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:40 dist-test-slave-cbn3 krb5kdc[467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980660, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:40.917824 480 init.cc:377] Logged in from keytab as kudu/127.10.131.62@KRBTEST.COM (short username kudu)
I20260620 18:37:40.919996 480 webserver.cc:460] Webserver started at http://127.10.131.62:35279/ using document root <none> and password file <none>
I20260620 18:37:40.920786 480 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:40.920985 480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:40.921332 480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:40.925431 480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "4cd41f9f9d7744bba4cd088ab2256745"
format_stamp: "Formatted at 2026-06-20 18:37:40 on dist-test-slave-cbn3"
server_key: "78fe3fc1231e035f0e0015ee1bee15e5"
server_key_iv: "5ffa882521560f36118fb3e21cf1c1d1"
server_key_version: "encryptionkey@0"
I20260620 18:37:40.926293 480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "4cd41f9f9d7744bba4cd088ab2256745"
format_stamp: "Formatted at 2026-06-20 18:37:40 on dist-test-slave-cbn3"
server_key: "78fe3fc1231e035f0e0015ee1bee15e5"
server_key_iv: "5ffa882521560f36118fb3e21cf1c1d1"
server_key_version: "encryptionkey@0"
I20260620 18:37:40.931977 480 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 18:37:40.935702 495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:40.937202 480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:37:40.937399 480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "4cd41f9f9d7744bba4cd088ab2256745"
format_stamp: "Formatted at 2026-06-20 18:37:40 on dist-test-slave-cbn3"
server_key: "78fe3fc1231e035f0e0015ee1bee15e5"
server_key_iv: "5ffa882521560f36118fb3e21cf1c1d1"
server_key_version: "encryptionkey@0"
I20260620 18:37:40.937610 480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:40.953018 480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:40.957592 480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:40.957907 480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:40.976639 480 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:45033
I20260620 18:37:40.976660 546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:45033 every 8 connection(s)
I20260620 18:37:40.978084 480 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:37:40.981871 547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:40.986035 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 480
I20260620 18:37:40.986244 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:37:40.986720 10764 external_mini_cluster.cc:1428] Setting key 52d415eb09342975242a3fc431c43fcf
I20260620 18:37:40.996048 547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cd41f9f9d7744bba4cd088ab2256745: Bootstrap starting.
I20260620 18:37:40.999814 547 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cd41f9f9d7744bba4cd088ab2256745: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:41.001088 547 log.cc:824] T 00000000000000000000000000000000 P 4cd41f9f9d7744bba4cd088ab2256745: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:41.001663 550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:40.988778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53247 (local address 127.10.131.62:45033)
0620 18:37:40.989679 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:40.989694 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:40.989745 (+ 51us) server_negotiation.cc:407] Connection header received
0620 18:37:40.991492 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:40.991525 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:40.992102 (+ 577us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:40.992574 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:40.993142 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:40.995130 (+ 1988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:40.996558 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:40.997561 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:40.997776 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:40.998876 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:40.998889 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:40.998907 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:40.999619 (+ 712us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:40.999719 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:40.999754 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:40.999889 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:41.000282 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 18:37:41.000538 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":344,"threads_started":1}
I20260620 18:37:41.004637 547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cd41f9f9d7744bba4cd088ab2256745: No bootstrap required, opened a new log
I20260620 18:37:41.006250 10764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:37:41.016702 547 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cd41f9f9d7744bba4cd088ab2256745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd41f9f9d7744bba4cd088ab2256745" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45033 } }
I20260620 18:37:41.017284 547 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4cd41f9f9d7744bba4cd088ab2256745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:41.017432 547 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cd41f9f9d7744bba4cd088ab2256745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd41f9f9d7744bba4cd088ab2256745, State: Initialized, Role: FOLLOWER
I20260620 18:37:41.018028 547 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cd41f9f9d7744bba4cd088ab2256745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd41f9f9d7744bba4cd088ab2256745" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45033 } }
I20260620 18:37:41.018265 547 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4cd41f9f9d7744bba4cd088ab2256745 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:37:41.018410 547 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cd41f9f9d7744bba4cd088ab2256745 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:37:41.018601 547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cd41f9f9d7744bba4cd088ab2256745 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:41.020814 547 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cd41f9f9d7744bba4cd088ab2256745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd41f9f9d7744bba4cd088ab2256745" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45033 } }
I20260620 18:37:41.021428 547 leader_election.cc:302] T 00000000000000000000000000000000 P 4cd41f9f9d7744bba4cd088ab2256745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cd41f9f9d7744bba4cd088ab2256745; no voters:
I20260620 18:37:41.022085 547 leader_election.cc:288] T 00000000000000000000000000000000 P 4cd41f9f9d7744bba4cd088ab2256745 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:37:41.022256 557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cd41f9f9d7744bba4cd088ab2256745 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:37:41.022811 550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:41.015498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37711 (local address 127.10.131.62:45033)
0620 18:37:41.015751 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:41.015758 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:41.016055 (+ 297us) server_negotiation.cc:407] Connection header received
0620 18:37:41.016249 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:41.016253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:41.016322 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:41.016414 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:41.017140 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:41.018212 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:41.019756 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:41.020276 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:41.020422 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:41.021747 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:41.021752 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:41.021755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:41.021774 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:41.021882 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:41.021914 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:41.022030 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:41.022222 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 18:37:41.022338 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 18:37:41.023125 557 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cd41f9f9d7744bba4cd088ab2256745 [term 1 LEADER]: Becoming Leader. State: Replica: 4cd41f9f9d7744bba4cd088ab2256745, State: Running, Role: LEADER
I20260620 18:37:41.023797 557 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cd41f9f9d7744bba4cd088ab2256745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd41f9f9d7744bba4cd088ab2256745" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45033 } }
I20260620 18:37:41.026080 558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cd41f9f9d7744bba4cd088ab2256745 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cd41f9f9d7744bba4cd088ab2256745. Latest consensus state: current_term: 1 leader_uuid: "4cd41f9f9d7744bba4cd088ab2256745" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd41f9f9d7744bba4cd088ab2256745" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45033 } } }
I20260620 18:37:41.026521 558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cd41f9f9d7744bba4cd088ab2256745 [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:41.026527 547 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cd41f9f9d7744bba4cd088ab2256745 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:37:41.027799 561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:37:41.027685 559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cd41f9f9d7744bba4cd088ab2256745 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cd41f9f9d7744bba4cd088ab2256745" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd41f9f9d7744bba4cd088ab2256745" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 45033 } } }
I20260620 18:37:41.028016 559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cd41f9f9d7744bba4cd088ab2256745 [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:41.033463 561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:37:41.042933 561 catalog_manager.cc:1347] Generated new cluster ID: 6c929d2318e141f5b32fa008725e16d7
I20260620 18:37:41.043073 561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:37:41.050244 561 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:37:41.051626 561 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:37:41.064889 561 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4cd41f9f9d7744bba4cd088ab2256745: Generated new TSK 0
I20260620 18:37:41.065812 561 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:37:41.077879 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 480
W20260620 18:37:41.079041 542 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53247
W20260620 18:37:41.079726 10764 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 480: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:37:41.079877 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 480
2026-06-20T18:37:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1903 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:37:41 dist-test-slave-cbn3 krb5kdc[578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:37:41 dist-test-slave-cbn3 krb5kdc[578](info): set up 2 sockets
Jun 20 18:37:41 dist-test-slave-cbn3 krb5kdc[578](info): commencing operation
krb5kdc: starting...
W20260620 18:37:42.186074 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:37:42 dist-test-slave-cbn3 krb5kdc[578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:37:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:37:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.131.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
I20260620 18:37:42.445354 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:38593
--webserver_interface=127.10.131.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:43009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:38593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:42.802942 591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:42.803310 591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:42.803496 591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:42.811471 591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:37:42.811647 591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:42.811764 591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:42.811875 591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:37:42.811960 591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:37:42.822237 591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:43009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:38593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:38593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:37:42.823592 591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:42.825037 591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:42.837389 596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:42.837410 597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:42.837347 599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:42.838227 591 server_base.cc:1034] running on GCE node
I20260620 18:37:42.839082 591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:42.840590 591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:42.841845 591 hybrid_clock.cc:648] HybridClock initialized: now 1781980662841786 us; error 76 us; skew 500 ppm
I20260620 18:37:42.842213 591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:42 dist-test-slave-cbn3 krb5kdc[578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980662, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:42.845491 591 init.cc:377] Logged in from keytab as kudu/127.10.131.62@KRBTEST.COM (short username kudu)
I20260620 18:37:42.846998 591 webserver.cc:460] Webserver started at http://127.10.131.62:43725/ using document root <none> and password file <none>
I20260620 18:37:42.847669 591 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:42.847774 591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:42.848033 591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:42.851011 591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "911ef88988c7489b8440d1bddca55ce4"
format_stamp: "Formatted at 2026-06-20 18:37:42 on dist-test-slave-cbn3"
server_key: "e650c275d220598a52105733bfa35736"
server_key_iv: "801cf1e7c1ecdd4ab1f9fc9e2d5ba8b3"
server_key_version: "encryptionkey@0"
I20260620 18:37:42.851715 591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "911ef88988c7489b8440d1bddca55ce4"
format_stamp: "Formatted at 2026-06-20 18:37:42 on dist-test-slave-cbn3"
server_key: "e650c275d220598a52105733bfa35736"
server_key_iv: "801cf1e7c1ecdd4ab1f9fc9e2d5ba8b3"
server_key_version: "encryptionkey@0"
I20260620 18:37:42.857122 591 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 18:37:42.860499 606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:42.861869 591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:37:42.862046 591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "911ef88988c7489b8440d1bddca55ce4"
format_stamp: "Formatted at 2026-06-20 18:37:42 on dist-test-slave-cbn3"
server_key: "e650c275d220598a52105733bfa35736"
server_key_iv: "801cf1e7c1ecdd4ab1f9fc9e2d5ba8b3"
server_key_version: "encryptionkey@0"
I20260620 18:37:42.862224 591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:42.867897 591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:42.871791 591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:42.872094 591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:42.890053 591 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:38593
I20260620 18:37:42.890089 657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:38593 every 8 connection(s)
I20260620 18:37:42.891484 591 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:37:42.895300 658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:42.900817 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 591
I20260620 18:37:42.901067 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:37:42.901525 10764 external_mini_cluster.cc:1428] Setting key cc7ae85ff80a73a0783a7d1995897d1c
I20260620 18:37:42.906796 658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 911ef88988c7489b8440d1bddca55ce4: Bootstrap starting.
I20260620 18:37:42.910516 661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:42.903378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55231 (local address 127.10.131.62:38593)
0620 18:37:42.904278 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:42.904294 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:42.904346 (+ 52us) server_negotiation.cc:407] Connection header received
0620 18:37:42.906681 (+ 2335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:42.906709 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:42.907142 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:42.907584 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:42.908583 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:42.908595 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:42.908615 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:42.908640 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:42.908708 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:42.908742 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:42.908997 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:42.909127 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 18:37:42.909319 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":367,"threads_started":1}
I20260620 18:37:42.910880 658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 911ef88988c7489b8440d1bddca55ce4: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:42.911985 658 log.cc:824] T 00000000000000000000000000000000 P 911ef88988c7489b8440d1bddca55ce4: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:42.914371 10764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:37:42.915122 658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 911ef88988c7489b8440d1bddca55ce4: No bootstrap required, opened a new log
I20260620 18:37:42.921422 661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:42.918785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46901 (local address 127.10.131.62:38593)
0620 18:37:42.919018 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:42.919027 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:42.919432 (+ 405us) server_negotiation.cc:407] Connection header received
0620 18:37:42.919592 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:42.919596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:42.919648 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:42.919735 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:42.920714 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:42.920718 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:42.920721 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:37:42.920736 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:42.920774 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:37:42.920792 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:42.920959 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:42.921057 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 18:37:42.921105 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:37:42.923880 658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 911ef88988c7489b8440d1bddca55ce4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911ef88988c7489b8440d1bddca55ce4" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38593 } }
I20260620 18:37:42.924273 658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 911ef88988c7489b8440d1bddca55ce4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:37:42.924403 658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 911ef88988c7489b8440d1bddca55ce4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 911ef88988c7489b8440d1bddca55ce4, State: Initialized, Role: FOLLOWER
I20260620 18:37:42.925081 658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 911ef88988c7489b8440d1bddca55ce4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911ef88988c7489b8440d1bddca55ce4" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38593 } }
I20260620 18:37:42.925350 658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 911ef88988c7489b8440d1bddca55ce4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:37:42.925504 658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 911ef88988c7489b8440d1bddca55ce4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:37:42.925724 658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 911ef88988c7489b8440d1bddca55ce4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:42.927949 658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 911ef88988c7489b8440d1bddca55ce4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911ef88988c7489b8440d1bddca55ce4" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38593 } }
I20260620 18:37:42.928459 658 leader_election.cc:302] T 00000000000000000000000000000000 P 911ef88988c7489b8440d1bddca55ce4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 911ef88988c7489b8440d1bddca55ce4; no voters:
I20260620 18:37:42.929108 658 leader_election.cc:288] T 00000000000000000000000000000000 P 911ef88988c7489b8440d1bddca55ce4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:37:42.929239 668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 911ef88988c7489b8440d1bddca55ce4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:37:42.929939 668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 911ef88988c7489b8440d1bddca55ce4 [term 1 LEADER]: Becoming Leader. State: Replica: 911ef88988c7489b8440d1bddca55ce4, State: Running, Role: LEADER
I20260620 18:37:42.930531 668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 911ef88988c7489b8440d1bddca55ce4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911ef88988c7489b8440d1bddca55ce4" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38593 } }
I20260620 18:37:42.931447 658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 911ef88988c7489b8440d1bddca55ce4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:37:42.933274 669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 911ef88988c7489b8440d1bddca55ce4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "911ef88988c7489b8440d1bddca55ce4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911ef88988c7489b8440d1bddca55ce4" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38593 } } }
I20260620 18:37:42.933728 670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 911ef88988c7489b8440d1bddca55ce4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 911ef88988c7489b8440d1bddca55ce4. Latest consensus state: current_term: 1 leader_uuid: "911ef88988c7489b8440d1bddca55ce4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911ef88988c7489b8440d1bddca55ce4" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 38593 } } }
I20260620 18:37:42.934002 669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 911ef88988c7489b8440d1bddca55ce4 [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:42.934065 670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 911ef88988c7489b8440d1bddca55ce4 [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:42.938323 674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:37:42.943665 674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:37:42.953577 674 catalog_manager.cc:1347] Generated new cluster ID: 092f487ef6c84e57a1fcdd63fcd6ccba
I20260620 18:37:42.953714 674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:37:42.963376 674 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:37:42.963775 674 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:37:42.977090 674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 911ef88988c7489b8440d1bddca55ce4: Generated new TSK 0
I20260620 18:37:42.977903 674 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:37:42.988266 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 591
W20260620 18:37:42.989962 653 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55231
W20260620 18:37:42.990639 10764 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 591: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:37:42.990796 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 591
2026-06-20T18:37:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1907 ms)
[----------] 18 tests from AuthTokenIssuingTest (34494 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:37:43 dist-test-slave-cbn3 krb5kdc[689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:37:43 dist-test-slave-cbn3 krb5kdc[689](info): set up 2 sockets
Jun 20 18:37:43 dist-test-slave-cbn3 krb5kdc[689](info): commencing operation
krb5kdc: starting...
W20260620 18:37:44.101624 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:37:44 dist-test-slave-cbn3 krb5kdc[689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:37:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:37:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.131.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
I20260620 18:37:44.354439 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:34293
--webserver_interface=127.10.131.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:34963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:34293,127.10.131.61:35945,127.10.131.60:44749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:44.683511 702 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:37:44.683885 702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:44.684020 702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:44.684149 702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:44.691736 702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:37:44.691871 702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:44.691992 702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:44.692080 702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:37:44.692193 702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:37:44.702646 702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:34963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:34293,127.10.131.61:35945,127.10.131.60:44749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:34293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:37:44.704008 702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:44.705425 702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:44.717564 707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:44.717540 710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:44.717486 708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:44.718282 702 server_base.cc:1034] running on GCE node
I20260620 18:37:44.719126 702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:44.720496 702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:44.721729 702 hybrid_clock.cc:648] HybridClock initialized: now 1781980664721676 us; error 65 us; skew 500 ppm
I20260620 18:37:44.722061 702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:44 dist-test-slave-cbn3 krb5kdc[689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980664, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:44.729768 702 init.cc:377] Logged in from keytab as kudu/127.10.131.62@KRBTEST.COM (short username kudu)
I20260620 18:37:44.731484 702 webserver.cc:460] Webserver started at http://127.10.131.62:38041/ using document root <none> and password file <none>
I20260620 18:37:44.732273 702 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:44.732435 702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:44.732831 702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:44.736274 702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "0f59e97528014856b2665408eb4c66a0"
format_stamp: "Formatted at 2026-06-20 18:37:44 on dist-test-slave-cbn3"
server_key: "e583001cf638ad774bcef2d50681e4fc"
server_key_iv: "0e4b65ac1b3835841efec318ad223126"
server_key_version: "encryptionkey@0"
I20260620 18:37:44.737335 702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "0f59e97528014856b2665408eb4c66a0"
format_stamp: "Formatted at 2026-06-20 18:37:44 on dist-test-slave-cbn3"
server_key: "e583001cf638ad774bcef2d50681e4fc"
server_key_iv: "0e4b65ac1b3835841efec318ad223126"
server_key_version: "encryptionkey@0"
I20260620 18:37:44.743245 702 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:37:44.746827 717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:44.748168 702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:37:44.748356 702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "0f59e97528014856b2665408eb4c66a0"
format_stamp: "Formatted at 2026-06-20 18:37:44 on dist-test-slave-cbn3"
server_key: "e583001cf638ad774bcef2d50681e4fc"
server_key_iv: "0e4b65ac1b3835841efec318ad223126"
server_key_version: "encryptionkey@0"
I20260620 18:37:44.748587 702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:44.772962 702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:44.777400 702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:44.777709 702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:44.795955 702 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:34293
I20260620 18:37:44.795974 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:34293 every 8 connection(s)
I20260620 18:37:44.797298 702 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:37:44.799978 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 702
I20260620 18:37:44.800199 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:37:44.801065 10764 external_mini_cluster.cc:1428] Setting key cfa92a36dc12875d61e4d8ff2cabced6
I20260620 18:37:44.801465 769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:44.811273 769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34293 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:37:44 dist-test-slave-cbn3 krb5kdc[689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980664, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.131.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.61@KRBTEST.COM" created.
I20260620 18:37:44.833011 773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:44.814438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43760 (local address 127.10.131.62:34293)
0620 18:37:44.816067 (+ 1629us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:44.816075 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:44.816122 (+ 47us) server_negotiation.cc:407] Connection header received
0620 18:37:44.817706 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:44.817712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:44.817976 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:44.818166 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:44.819385 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:44.820562 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:44.821960 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:44.822479 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:44.822666 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:44.826088 (+ 3422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:44.826096 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:44.826106 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:44.826131 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:44.828810 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:44.829595 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:44.829603 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:44.829607 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:44.829744 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:44.830132 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:44.830136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:44.830139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:44.830593 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:44.830789 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:44.831265 (+ 476us) server_negotiation.cc:299] Negotiation successful
0620 18:37:44.831580 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":91,"server-negotiator.queue_time_us":1402,"thread_start_us":1158,"threads_started":1}
I20260620 18:37:44.833048 772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:44.814438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:34293 (local address 127.0.0.1:43760)
0620 18:37:44.815222 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:44.815273 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:44.817343 (+ 2070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:44.818181 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:44.818187 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:44.818884 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:44.819186 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:44.819201 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:44.820720 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:44.820724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:44.821832 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:44.821845 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:44.822704 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:44.822710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:44.822985 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:44.823740 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:44.823760 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:44.825851 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:44.829029 (+ 3178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:44.829040 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:44.829050 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:44.829471 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:44.829850 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:44.829855 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:44.829859 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:44.830007 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:44.830721 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:44.830728 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:44.830918 (+ 190us) client_negotiation.cc:769] Sending connection context
0620 18:37:44.831145 (+ 227us) client_negotiation.cc:240] Negotiation successful
0620 18:37:44.831686 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":191,"thread_start_us":335,"threads_started":1}
I20260620 18:37:44.835930 769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.131.61" port: 35945 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:37:44.836952 772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:44.836515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:35945 (local address 127.0.0.1:56228)
0620 18:37:44.836675 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:44.836784 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.61:35945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 18:37:44.837162 718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.131.61:35945: connect: Connection refused (error 111)
I20260620 18:37:44.838631 772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:44.838299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:35945 (local address 127.0.0.1:56242)
0620 18:37:44.838438 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:44.838510 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.61:35945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 18:37:44.838795 769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.61:35945: Network error: Client connection negotiation failed: client connection to 127.10.131.61:35945: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.61.keytab.
Entry for principal kudu/127.10.131.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.61.keytab.
I20260620 18:37:44.857934 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.61:35945
--webserver_interface=127.10.131.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.61.keytab
--principal=kudu/127.10.131.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:34963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:34293,127.10.131.61:35945,127.10.131.60:44749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:37:44.888129 769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.61" port: 35945 } attempt: 1
I20260620 18:37:44.889370 772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:44.888813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:35945 (local address 127.0.0.1:56250)
0620 18:37:44.889045 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:44.889198 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.61:35945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 18:37:44.890419 772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:44.890021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:35945 (local address 127.0.0.1:56252)
0620 18:37:44.890178 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:44.890263 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.61:35945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 18:37:44.890630 769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.61:35945: Network error: Client connection negotiation failed: client connection to 127.10.131.61:35945: connect: Connection refused (error 111)
I20260620 18:37:44.958896 769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.61" port: 35945 } attempt: 2
I20260620 18:37:44.960112 772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:44.959549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:35945 (local address 127.0.0.1:56268)
0620 18:37:44.959837 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:44.959946 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.61:35945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 18:37:44.961169 772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:44.960719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:35945 (local address 127.0.0.1:56270)
0620 18:37:44.960923 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:44.961004 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.61:35945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 18:37:44.961352 769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.61:35945: Network error: Client connection negotiation failed: client connection to 127.10.131.61:35945: connect: Connection refused (error 111)
I20260620 18:37:45.052600 769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.61" port: 35945 } attempt: 3
I20260620 18:37:45.053746 772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.053352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:35945 (local address 127.0.0.1:56280)
0620 18:37:45.053531 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:45.053618 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.61:35945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 18:37:45.054741 772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.054450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:35945 (local address 127.0.0.1:56286)
0620 18:37:45.054566 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:45.054625 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.61:35945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 18:37:45.055008 769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.61:35945: Network error: Client connection negotiation failed: client connection to 127.10.131.61:35945: connect: Connection refused (error 111)
W20260620 18:37:45.183364 776 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:37:45.183707 776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:45.183823 776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:45.183966 776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:45.191285 776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:37:45.191383 776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:45.191481 776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:45.191568 776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:37:45.191677 776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:37:45.198331 769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.61" port: 35945 } attempt: 4
I20260620 18:37:45.199643 772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.199035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:35945 (local address 127.0.0.1:56298)
0620 18:37:45.199426 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:45.199514 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.61:35945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260620 18:37:45.200639 772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.200275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:35945 (local address 127.0.0.1:56312)
0620 18:37:45.200418 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:45.200532 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.61:35945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 18:37:45.200860 769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.61:35945: Network error: Client connection negotiation failed: client connection to 127.10.131.61:35945: connect: Connection refused (error 111)
I20260620 18:37:45.202293 776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:34963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:34293,127.10.131.61:35945,127.10.131.60:44749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.61.keytab
--principal=kudu/127.10.131.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.61:35945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.131.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:37:45.203568 776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:45.204891 776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:45.216619 782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:45.216638 784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:45.217221 781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:45.217442 776 server_base.cc:1034] running on GCE node
I20260620 18:37:45.218238 776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:45.219538 776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:45.220772 776 hybrid_clock.cc:648] HybridClock initialized: now 1781980665220707 us; error 43 us; skew 500 ppm
I20260620 18:37:45.221138 776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:45 dist-test-slave-cbn3 krb5kdc[689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980665, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:45.224210 776 init.cc:377] Logged in from keytab as kudu/127.10.131.61@KRBTEST.COM (short username kudu)
I20260620 18:37:45.225694 776 webserver.cc:460] Webserver started at http://127.10.131.61:43511/ using document root <none> and password file <none>
I20260620 18:37:45.226418 776 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:45.226550 776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:45.226866 776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:45.229928 776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-1/data/instance:
uuid: "18a834b0ead34344826f142564fe8858"
format_stamp: "Formatted at 2026-06-20 18:37:45 on dist-test-slave-cbn3"
server_key: "747d94884ba74a95b2a0afe375fe8f95"
server_key_iv: "a6879d9ff324bbf0b01bcb5d771071f8"
server_key_version: "encryptionkey@0"
I20260620 18:37:45.230793 776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-1/wal/instance:
uuid: "18a834b0ead34344826f142564fe8858"
format_stamp: "Formatted at 2026-06-20 18:37:45 on dist-test-slave-cbn3"
server_key: "747d94884ba74a95b2a0afe375fe8f95"
server_key_iv: "a6879d9ff324bbf0b01bcb5d771071f8"
server_key_version: "encryptionkey@0"
I20260620 18:37:45.235893 776 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:37:45.239719 791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:45.241048 776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:37:45.241230 776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-1/wal
uuid: "18a834b0ead34344826f142564fe8858"
format_stamp: "Formatted at 2026-06-20 18:37:45 on dist-test-slave-cbn3"
server_key: "747d94884ba74a95b2a0afe375fe8f95"
server_key_iv: "a6879d9ff324bbf0b01bcb5d771071f8"
server_key_version: "encryptionkey@0"
I20260620 18:37:45.241451 776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:45.257838 776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:45.261674 776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:45.261931 776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:45.279145 776 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.61:35945
I20260620 18:37:45.279165 842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.61:35945 every 8 connection(s)
I20260620 18:37:45.280434 776 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-1/data/info.pb
I20260620 18:37:45.281625 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 776
I20260620 18:37:45.281800 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-1/wal/instance
I20260620 18:37:45.282231 10764 external_mini_cluster.cc:1428] Setting key 5e57bea2618d60bf988a85c95fd4a5bf
I20260620 18:37:45.284243 843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:45.291127 843 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34293 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.131.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.60@KRBTEST.COM" created.
Jun 20 18:37:45 dist-test-slave-cbn3 krb5kdc[689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980665, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.61@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:37:45.314320 773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.293763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43774 (local address 127.10.131.62:34293)
0620 18:37:45.294019 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:45.294028 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:45.295091 (+ 1063us) server_negotiation.cc:407] Connection header received
0620 18:37:45.297000 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:45.297009 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:45.297100 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:45.297230 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:45.298624 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:45.299842 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:45.301507 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:45.302294 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:45.302441 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:45.308078 (+ 5637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:45.308083 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:45.308086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:45.308121 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:45.310909 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:45.311685 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:45.311690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:45.311692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:45.311811 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:45.312345 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:45.312353 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:45.312359 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:45.312861 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:45.313022 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:45.313745 (+ 723us) server_negotiation.cc:299] Negotiation successful
0620 18:37:45.314013 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 18:37:45.315424 846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.294139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:34293 (local address 127.0.0.1:43774)
0620 18:37:45.294897 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:45.294946 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:45.296704 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:45.297418 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:45.297425 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:45.297903 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:45.298453 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:45.298470 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:45.299987 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:45.299991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:45.301350 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:45.301362 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:45.302492 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:45.302496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:45.302682 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:45.303367 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:45.303389 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:45.307832 (+ 4443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:45.311062 (+ 3230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:45.311072 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:45.311089 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:45.311543 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:45.311923 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:45.311928 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:45.311932 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:45.312114 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:45.312999 (+ 885us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:45.313005 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:45.313362 (+ 357us) client_negotiation.cc:769] Sending connection context
0620 18:37:45.313585 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 18:37:45.313974 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":327,"threads_started":1}
I20260620 18:37:45.317224 843 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.131.61" port: 35945 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:37:45 dist-test-slave-cbn3 krb5kdc[689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980665, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.61@KRBTEST.COM for kudu/127.10.131.61@KRBTEST.COM
I20260620 18:37:45.332276 846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.317944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:35945 (local address 127.0.0.1:56326)
0620 18:37:45.318142 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:45.318163 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:45.318264 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:45.319043 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:45.319051 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:45.319549 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:45.319778 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:45.319792 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:45.321192 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:45.321200 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:45.322769 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:45.322785 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:45.323552 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:45.323560 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:45.323787 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:45.324374 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:45.324400 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:45.326714 (+ 2314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:45.329888 (+ 3174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:45.329893 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:45.329897 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:45.330250 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:45.330634 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:45.330641 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:45.330647 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:45.330786 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:45.331344 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:45.331349 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:45.331476 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 18:37:45.331620 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 18:37:45.331876 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 18:37:45.332391 848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.318054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56326 (local address 127.10.131.61:35945)
0620 18:37:45.318652 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:45.318658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:45.318681 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:37:45.318722 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:45.318727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:45.318887 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:45.318995 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:45.319959 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:45.321042 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:45.322910 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:45.323407 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:45.323537 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:45.326914 (+ 3377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:45.326923 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:45.326935 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:45.326960 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:45.329749 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:45.330389 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:45.330401 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:45.330403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:45.330515 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:45.330885 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:45.330892 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:45.330895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:45.331211 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:45.331409 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:45.331770 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 18:37:45.331985 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":328,"threads_started":1}
I20260620 18:37:45.334908 843 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.131.60" port: 44749 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:37:45.335950 846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.335468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:44749 (local address 127.0.0.1:53832)
0620 18:37:45.335619 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:45.335765 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:44749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 18:37:45.336165 792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.131.60:44749: connect: Connection refused (error 111)
I20260620 18:37:45.337615 846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.337276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:44749 (local address 127.0.0.1:53842)
0620 18:37:45.337446 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:45.337522 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:44749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 18:37:45.337839 843 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.60:44749: Network error: Client connection negotiation failed: client connection to 127.10.131.60:44749: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.60.keytab.
Entry for principal kudu/127.10.131.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.60.keytab.
I20260620 18:37:45.339440 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.60:44749
--webserver_interface=127.10.131.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.60.keytab
--principal=kudu/127.10.131.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:34963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:34293,127.10.131.61:35945,127.10.131.60:44749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:37:45.387156 843 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.60" port: 44749 } attempt: 1
I20260620 18:37:45.388458 846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.387994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:44749 (local address 127.0.0.1:53856)
0620 18:37:45.388202 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:45.388306 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:44749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 18:37:45.389490 846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.389135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:44749 (local address 127.0.0.1:53868)
0620 18:37:45.389312 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:45.389388 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:44749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 18:37:45.389698 843 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.60:44749: Network error: Client connection negotiation failed: client connection to 127.10.131.60:44749: connect: Connection refused (error 111)
I20260620 18:37:45.457888 843 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.60" port: 44749 } attempt: 2
I20260620 18:37:45.458981 846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.458542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:44749 (local address 127.0.0.1:53880)
0620 18:37:45.458736 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:45.458831 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:44749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 18:37:45.459913 846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.459601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:44749 (local address 127.0.0.1:53886)
0620 18:37:45.459736 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:45.459805 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:44749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 18:37:45.460127 843 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.60:44749: Network error: Client connection negotiation failed: client connection to 127.10.131.60:44749: connect: Connection refused (error 111)
I20260620 18:37:45.500124 769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.61" port: 35945 } attempt: 5
Jun 20 18:37:45 dist-test-slave-cbn3 krb5kdc[689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980664, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for kudu/127.10.131.61@KRBTEST.COM
I20260620 18:37:45.515323 772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.500928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:35945 (local address 127.0.0.1:56328)
0620 18:37:45.501164 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:45.501185 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:45.501314 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:45.501704 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:45.501707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:45.502046 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:45.502217 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:45.502226 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:45.503600 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:45.503603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:45.504444 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:45.504455 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:45.505446 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:45.505450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:45.505620 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:45.506357 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:45.506381 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:45.508937 (+ 2556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:45.513070 (+ 4133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:45.513074 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:45.513076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:45.513432 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:45.513851 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:45.513854 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:45.513856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:45.513997 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:45.514585 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:45.514588 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:45.514713 (+ 125us) client_negotiation.cc:769] Sending connection context
0620 18:37:45.514825 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 18:37:45.515087 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 18:37:45.515514 848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.501149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56328 (local address 127.10.131.61:35945)
0620 18:37:45.501352 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:45.501358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:45.501402 (+ 44us) server_negotiation.cc:407] Connection header received
0620 18:37:45.501466 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:45.501472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:45.501567 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:45.501671 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:45.502366 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:45.503471 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:45.504598 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:45.505310 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:45.505438 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:45.509143 (+ 3705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:45.509152 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:45.509169 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:45.509220 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:45.512920 (+ 3700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:45.513554 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:45.513559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:45.513583 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:45.513731 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:45.514095 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:45.514099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:45.514102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:45.514477 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:45.514623 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:45.514946 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 18:37:45.515206 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:37:45.516590 769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.131.60" port: 44749 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:37:45.517620 772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.517194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:44749 (local address 127.0.0.1:53898)
0620 18:37:45.517368 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:45.517458 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:44749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 18:37:45.518636 772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.518196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:44749 (local address 127.0.0.1:53906)
0620 18:37:45.518360 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:45.518490 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:44749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 18:37:45.518865 769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.60:44749: Network error: Client connection negotiation failed: client connection to 127.10.131.60:44749: connect: Connection refused (error 111)
I20260620 18:37:45.551309 843 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.60" port: 44749 } attempt: 3
I20260620 18:37:45.552399 846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.551949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:44749 (local address 127.0.0.1:53920)
0620 18:37:45.552131 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:45.552232 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:44749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 18:37:45.553514 846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.553118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:44749 (local address 127.0.0.1:53924)
0620 18:37:45.553328 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:45.553410 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:44749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260620 18:37:45.553705 843 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.60:44749: Network error: Client connection negotiation failed: client connection to 127.10.131.60:44749: connect: Connection refused (error 111)
I20260620 18:37:45.570101 769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.60" port: 44749 } attempt: 1
I20260620 18:37:45.571329 772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.570759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:44749 (local address 127.0.0.1:53940)
0620 18:37:45.570997 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:45.571114 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:44749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 18:37:45.572399 772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.571954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:44749 (local address 127.0.0.1:53946)
0620 18:37:45.572141 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:45.572240 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:44749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 18:37:45.572600 769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.60:44749: Network error: Client connection negotiation failed: client connection to 127.10.131.60:44749: connect: Connection refused (error 111)
I20260620 18:37:45.640825 769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.60" port: 44749 } attempt: 2
I20260620 18:37:45.642107 772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.641526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:44749 (local address 127.0.0.1:53958)
0620 18:37:45.641767 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:45.641894 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:44749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 18:37:45.643169 772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.642735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:44749 (local address 127.0.0.1:53964)
0620 18:37:45.642908 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:45.643006 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:44749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 18:37:45.643386 769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.60:44749: Network error: Client connection negotiation failed: client connection to 127.10.131.60:44749: connect: Connection refused (error 111)
W20260620 18:37:45.673274 850 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:37:45.673560 850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:45.673630 850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:45.673717 850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:45.680647 850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:37:45.680733 850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:45.680789 850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:45.680851 850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:37:45.680907 850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:37:45.690189 850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:34963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:34293,127.10.131.61:35945,127.10.131.60:44749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.60.keytab
--principal=kudu/127.10.131.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.60:44749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.131.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:37:45.691431 850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:45.692646 850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:37:45.696952 843 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.60" port: 44749 } attempt: 4
I20260620 18:37:45.698304 846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.697622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:44749 (local address 127.0.0.1:53966)
0620 18:37:45.697976 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:45.698088 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:44749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260620 18:37:45.699512 846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.699065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:44749 (local address 127.0.0.1:53976)
0620 18:37:45.699287 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:45.699390 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:44749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260620 18:37:45.699747 843 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.60:44749: Network error: Client connection negotiation failed: client connection to 127.10.131.60:44749: connect: Connection refused (error 111)
W20260620 18:37:45.705497 855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:45.705464 856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:45.705523 858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:45.706246 850 server_base.cc:1034] running on GCE node
I20260620 18:37:45.706933 850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:45.708222 850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:45.709444 850 hybrid_clock.cc:648] HybridClock initialized: now 1781980665709383 us; error 30 us; skew 500 ppm
I20260620 18:37:45.709722 850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:45 dist-test-slave-cbn3 krb5kdc[689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980665, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:45.712710 850 init.cc:377] Logged in from keytab as kudu/127.10.131.60@KRBTEST.COM (short username kudu)
I20260620 18:37:45.714231 850 webserver.cc:460] Webserver started at http://127.10.131.60:40801/ using document root <none> and password file <none>
I20260620 18:37:45.714908 850 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:45.715020 850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:45.715348 850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:45.718289 850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-2/data/instance:
uuid: "290918322a804cf3aa2d695379d92a29"
format_stamp: "Formatted at 2026-06-20 18:37:45 on dist-test-slave-cbn3"
server_key: "f239dfa938ceee970bf6c2a49c317809"
server_key_iv: "4a565b84507f077d95fb79fe955edf0c"
server_key_version: "encryptionkey@0"
I20260620 18:37:45.719084 850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-2/wal/instance:
uuid: "290918322a804cf3aa2d695379d92a29"
format_stamp: "Formatted at 2026-06-20 18:37:45 on dist-test-slave-cbn3"
server_key: "f239dfa938ceee970bf6c2a49c317809"
server_key_iv: "4a565b84507f077d95fb79fe955edf0c"
server_key_version: "encryptionkey@0"
I20260620 18:37:45.723980 850 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 18:37:45.727209 865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:45.728493 850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 18:37:45.728681 850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-2/wal
uuid: "290918322a804cf3aa2d695379d92a29"
format_stamp: "Formatted at 2026-06-20 18:37:45 on dist-test-slave-cbn3"
server_key: "f239dfa938ceee970bf6c2a49c317809"
server_key_iv: "4a565b84507f077d95fb79fe955edf0c"
server_key_version: "encryptionkey@0"
I20260620 18:37:45.728936 850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:45.747674 850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:45.749609 769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.60" port: 44749 } attempt: 3
I20260620 18:37:45.750779 772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.750188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:44749 (local address 127.0.0.1:53982)
0620 18:37:45.750420 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:45.750547 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:44749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 18:37:45.751843 772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.751518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:44749 (local address 127.0.0.1:53986)
0620 18:37:45.751654 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:45.751730 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:44749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 18:37:45.752169 769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.60:44749: Network error: Client connection negotiation failed: client connection to 127.10.131.60:44749: connect: Connection refused (error 111)
I20260620 18:37:45.752429 850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:45.752683 850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:45.771330 850 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.60:44749
I20260620 18:37:45.771354 916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.60:44749 every 8 connection(s)
I20260620 18:37:45.772733 850 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-2/data/info.pb
I20260620 18:37:45.773190 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 850
I20260620 18:37:45.773370 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781980314095189-10764-0/minicluster-data/master-2/wal/instance
I20260620 18:37:45.773911 10764 external_mini_cluster.cc:1428] Setting key d813f58312e4c4bd21dce88eb61b5223
I20260620 18:37:45.777374 917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:45.784299 917 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34293 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:37:45 dist-test-slave-cbn3 krb5kdc[689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.60@KRBTEST.COM
Jun 20 18:37:45 dist-test-slave-cbn3 krb5kdc[689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980665, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.60@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:37:45.796763 920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.775774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53992 (local address 127.10.131.60:44749)
0620 18:37:45.776518 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:45.776530 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:45.776572 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:37:45.778228 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:45.778250 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:45.778628 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:45.779005 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:45.780523 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:45.781904 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:45.783194 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:45.783849 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:45.784033 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:45.788557 (+ 4524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:45.788568 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:45.788577 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:45.788611 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:45.792003 (+ 3392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:45.792773 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:45.792781 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:45.792789 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:45.793064 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:45.793525 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:45.793531 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:45.793536 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:45.794093 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:45.794335 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:45.794846 (+ 511us) server_negotiation.cc:299] Negotiation successful
0620 18:37:45.795281 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":312,"threads_started":1}
I20260620 18:37:45.803817 773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.786978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43784 (local address 127.10.131.62:34293)
0620 18:37:45.787209 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:45.787217 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:45.787977 (+ 760us) server_negotiation.cc:407] Connection header received
0620 18:37:45.788154 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:45.788159 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:45.788223 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:45.788305 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:45.789693 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:45.790799 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:45.792290 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:45.792771 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:45.792953 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:45.796692 (+ 3739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:45.796699 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:45.796708 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:45.796748 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:45.800285 (+ 3537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:45.801380 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:45.801390 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:45.801393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:45.801555 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:45.802028 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:45.802033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:45.802036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:45.802436 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:45.802582 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:45.803215 (+ 633us) server_negotiation.cc:299] Negotiation successful
0620 18:37:45.803434 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:37:45.804589 921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.786963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:34293 (local address 127.0.0.1:43784)
0620 18:37:45.787813 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:45.787873 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:45.788038 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:45.788410 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:45.788419 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:45.789136 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:45.789529 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:45.789549 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:45.790958 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:45.790961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:45.792150 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:45.792164 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:45.792991 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:45.792996 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:45.793140 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:45.793783 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:45.793799 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:45.796481 (+ 2682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:45.800440 (+ 3959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:45.800456 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:45.800467 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:45.801220 (+ 753us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:45.801674 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:45.801682 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:45.801689 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:45.801904 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:45.802575 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:45.802586 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:45.802849 (+ 263us) client_negotiation.cc:769] Sending connection context
0620 18:37:45.803092 (+ 243us) client_negotiation.cc:240] Negotiation successful
0620 18:37:45.803542 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":426,"threads_started":1}
I20260620 18:37:45.806663 917 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.131.61" port: 35945 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:37:45 dist-test-slave-cbn3 krb5kdc[689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
Jun 20 18:37:45 dist-test-slave-cbn3 krb5kdc[689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980665, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.60@KRBTEST.COM for kudu/127.10.131.61@KRBTEST.COM
I20260620 18:37:45.819299 773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.800046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43786 (local address 127.10.131.62:34293)
0620 18:37:45.804123 (+ 4077us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:45.804129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:45.804145 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:37:45.804187 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:45.804193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:45.804255 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:45.804347 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:45.805995 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:45.807238 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:45.808608 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:45.809202 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:45.809288 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:45.813698 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:45.813706 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:45.813712 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:45.813750 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:45.816621 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:45.817427 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:45.817431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:45.817433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:45.817563 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:45.817967 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:45.817971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:45.817973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:45.818233 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:45.818510 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:45.818779 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 18:37:45.818980 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3970,"thread_start_us":6984,"threads_started":1}
I20260620 18:37:45.820365 921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.807538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:35945 (local address 127.0.0.1:56340)
0620 18:37:45.807705 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:45.807725 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:45.807836 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:45.808218 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:45.808222 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:45.808653 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:45.808874 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:45.808886 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:45.810112 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:45.810116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:45.811008 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:45.811020 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:45.811983 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:45.811987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:45.812122 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:45.812654 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:45.812670 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:45.814674 (+ 2004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:45.817954 (+ 3280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:45.817960 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:45.817964 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:45.818316 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:45.818701 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:45.818706 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:45.818710 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:45.818862 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:45.819449 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:45.819454 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:45.819605 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 18:37:45.819743 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 18:37:45.820017 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 18:37:45.820410 848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.807748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56340 (local address 127.10.131.61:35945)
0620 18:37:45.807952 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:45.807958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:45.807974 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:37:45.808031 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:45.808035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:45.808089 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:45.808198 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:45.809038 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:45.809966 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:45.811158 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:45.811841 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:45.811982 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:45.814858 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:45.814863 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:45.814866 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:45.814898 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:45.817819 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:45.818433 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:45.818440 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:45.818444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:45.818591 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:45.818967 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:45.818974 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:45.818977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:45.819299 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:45.819474 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:45.819828 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 18:37:45.820030 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:37:45.821341 917 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.131.60" port: 44749 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:37:45 dist-test-slave-cbn3 krb5kdc[689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980665, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.60@KRBTEST.COM for kudu/127.10.131.60@KRBTEST.COM
Jun 20 18:37:45 dist-test-slave-cbn3 krb5kdc[689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.61@KRBTEST.COM
I20260620 18:37:45.836270 921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.821917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:44749 (local address 127.0.0.1:53996)
0620 18:37:45.822056 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:45.822076 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:45.822195 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:45.822471 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:45.822476 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:45.822942 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:45.823119 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:45.823130 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:45.824434 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:45.824439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:45.825848 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:45.825861 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:45.826858 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:45.826864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:45.827041 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:45.827694 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:45.827716 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:45.830118 (+ 2402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:45.833617 (+ 3499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:45.833623 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:45.833627 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:45.834124 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:45.834486 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:45.834491 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:45.834495 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:45.834650 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:45.835236 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:45.835239 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:45.835404 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 18:37:45.835588 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 18:37:45.835861 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 18:37:45.836344 920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.822061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53996 (local address 127.10.131.60:44749)
0620 18:37:45.822206 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:45.822212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:45.822228 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:37:45.822294 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:45.822298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:45.822346 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:45.822415 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:45.823286 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:45.824299 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:45.825989 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:45.826674 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:45.826796 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:45.830279 (+ 3483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:45.830284 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:45.830287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:45.830313 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:45.833442 (+ 3129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:45.834231 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:45.834235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:45.834237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:45.834367 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:45.834758 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:45.834763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:45.834766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:45.835118 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:45.835267 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:45.835716 (+ 449us) server_negotiation.cc:299] Negotiation successful
0620 18:37:45.836048 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":48}
I20260620 18:37:45.837742 848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.821034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56344 (local address 127.10.131.61:35945)
0620 18:37:45.821186 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:45.821191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:45.821206 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:37:45.821360 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:45.821366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:45.821434 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:45.821531 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:45.823093 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:45.824198 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:45.825989 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:45.826684 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:45.826796 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:45.831609 (+ 4813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:45.831614 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:45.831617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:45.831644 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:45.834612 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:45.835378 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:45.835384 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:45.835387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:45.835541 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:45.835996 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:45.836002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:45.836005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:45.836334 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:45.836598 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:45.837189 (+ 591us) server_negotiation.cc:299] Negotiation successful
0620 18:37:45.837430 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:37:45.841748 917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 290918322a804cf3aa2d695379d92a29: Bootstrap starting.
I20260620 18:37:45.842435 10764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:37:45.845067 917 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 290918322a804cf3aa2d695379d92a29: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:45.846204 917 log.cc:824] T 00000000000000000000000000000000 P 290918322a804cf3aa2d695379d92a29: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:45.845108 744 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0f59e97528014856b2665408eb4c66a0"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43786
I20260620 18:37:45.849043 917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 290918322a804cf3aa2d695379d92a29: No bootstrap required, opened a new log
I20260620 18:37:45.856082 917 raft_consensus.cc:348] T 00000000000000000000000000000000 P 290918322a804cf3aa2d695379d92a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f59e97528014856b2665408eb4c66a0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34293 } } peers { permanent_uuid: "18a834b0ead34344826f142564fe8858" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 35945 } } peers { permanent_uuid: "290918322a804cf3aa2d695379d92a29" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 44749 } }
I20260620 18:37:45.856448 917 raft_consensus.cc:729] T 00000000000000000000000000000000 P 290918322a804cf3aa2d695379d92a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 290918322a804cf3aa2d695379d92a29, State: Initialized, Role: FOLLOWER
I20260620 18:37:45.856980 917 consensus_queue.cc:260] T 00000000000000000000000000000000 P 290918322a804cf3aa2d695379d92a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f59e97528014856b2665408eb4c66a0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34293 } } peers { permanent_uuid: "18a834b0ead34344826f142564fe8858" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 35945 } } peers { permanent_uuid: "290918322a804cf3aa2d695379d92a29" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 44749 } }
I20260620 18:37:45.857833 924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 290918322a804cf3aa2d695379d92a29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f59e97528014856b2665408eb4c66a0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34293 } } peers { permanent_uuid: "18a834b0ead34344826f142564fe8858" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 35945 } } peers { permanent_uuid: "290918322a804cf3aa2d695379d92a29" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 44749 } } }
I20260620 18:37:45.858162 924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 290918322a804cf3aa2d695379d92a29 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:37:45.859215 917 sys_catalog.cc:564] T 00000000000000000000000000000000 P 290918322a804cf3aa2d695379d92a29 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 18:37:45.868784 935 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 290918322a804cf3aa2d695379d92a29: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:37:45.868933 935 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:37:45.866757 850 master_runner.cc:186] Error getting master registration for 127.10.131.62:34293: OK, instance_id { permanent_uuid: "0f59e97528014856b2665408eb4c66a0" instance_seqno: 1781980664790700 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:37:45.875288 850 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:37:45.929423 769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.60" port: 44749 } attempt: 4
Jun 20 18:37:45 dist-test-slave-cbn3 krb5kdc[689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980664, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for kudu/127.10.131.60@KRBTEST.COM
I20260620 18:37:45.946125 772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.930257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:44749 (local address 127.0.0.1:54004)
0620 18:37:45.930494 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:45.930516 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:45.930660 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:45.931033 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:45.931040 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:45.931592 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:45.931806 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:45.931818 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:45.933343 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:45.933349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:45.934857 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:45.934872 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:45.935891 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:45.935897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:45.936106 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:45.936737 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:45.936766 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:45.939461 (+ 2695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:45.943327 (+ 3866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:45.943334 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:45.943338 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:45.943699 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:45.944068 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:45.944073 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:45.944077 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:45.944210 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:45.945059 (+ 849us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:45.945064 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:45.945255 (+ 191us) client_negotiation.cc:769] Sending connection context
0620 18:37:45.945419 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 18:37:45.945714 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 18:37:45.946229 920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.930321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54004 (local address 127.10.131.60:44749)
0620 18:37:45.930532 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:45.930537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:45.930592 (+ 55us) server_negotiation.cc:407] Connection header received
0620 18:37:45.930794 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:45.930799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:45.930878 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:45.930971 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:45.931983 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:45.933169 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:45.935005 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:45.935714 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:45.935849 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:45.939659 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:45.939664 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:45.939667 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:45.939698 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:45.943206 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:45.943821 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:45.943827 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:45.943833 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:45.943967 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:45.944337 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:45.944341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:45.944344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:45.944811 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:45.944999 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:45.945565 (+ 566us) server_negotiation.cc:299] Negotiation successful
0620 18:37:45.945857 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 18:37:45.952078 769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f59e97528014856b2665408eb4c66a0: Bootstrap starting.
I20260620 18:37:45.955556 769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f59e97528014856b2665408eb4c66a0: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:45.956598 769 log.cc:824] T 00000000000000000000000000000000 P 0f59e97528014856b2665408eb4c66a0: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:45.959550 769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f59e97528014856b2665408eb4c66a0: No bootstrap required, opened a new log
I20260620 18:37:45.965109 769 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f59e97528014856b2665408eb4c66a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f59e97528014856b2665408eb4c66a0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34293 } } peers { permanent_uuid: "18a834b0ead34344826f142564fe8858" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 35945 } } peers { permanent_uuid: "290918322a804cf3aa2d695379d92a29" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 44749 } }
I20260620 18:37:45.965485 769 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f59e97528014856b2665408eb4c66a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f59e97528014856b2665408eb4c66a0, State: Initialized, Role: FOLLOWER
I20260620 18:37:45.965901 769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f59e97528014856b2665408eb4c66a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f59e97528014856b2665408eb4c66a0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34293 } } peers { permanent_uuid: "18a834b0ead34344826f142564fe8858" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 35945 } } peers { permanent_uuid: "290918322a804cf3aa2d695379d92a29" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 44749 } }
I20260620 18:37:45.966811 938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f59e97528014856b2665408eb4c66a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f59e97528014856b2665408eb4c66a0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34293 } } peers { permanent_uuid: "18a834b0ead34344826f142564fe8858" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 35945 } } peers { permanent_uuid: "290918322a804cf3aa2d695379d92a29" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 44749 } } }
I20260620 18:37:45.967204 938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f59e97528014856b2665408eb4c66a0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:37:45.967957 769 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f59e97528014856b2665408eb4c66a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:37:45.968106 744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f59e97528014856b2665408eb4c66a0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 18:37:45.968357 744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f59e97528014856b2665408eb4c66a0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:45.972047 744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f59e97528014856b2665408eb4c66a0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f59e97528014856b2665408eb4c66a0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34293 } } peers { permanent_uuid: "18a834b0ead34344826f142564fe8858" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 35945 } } peers { permanent_uuid: "290918322a804cf3aa2d695379d92a29" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 44749 } }
I20260620 18:37:45.973706 744 leader_election.cc:288] T 00000000000000000000000000000000 P 0f59e97528014856b2665408eb4c66a0 [CANDIDATE]: Term 1 election: Requested vote from peers 18a834b0ead34344826f142564fe8858 (127.10.131.61:35945), 290918322a804cf3aa2d695379d92a29 (127.10.131.60:44749)
I20260620 18:37:45.974735 892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f59e97528014856b2665408eb4c66a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "290918322a804cf3aa2d695379d92a29"
I20260620 18:37:45.975230 892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 290918322a804cf3aa2d695379d92a29 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:45.978729 892 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 290918322a804cf3aa2d695379d92a29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f59e97528014856b2665408eb4c66a0 in term 1.
I20260620 18:37:45.979470 718 leader_election.cc:302] T 00000000000000000000000000000000 P 0f59e97528014856b2665408eb4c66a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f59e97528014856b2665408eb4c66a0, 290918322a804cf3aa2d695379d92a29; no voters:
I20260620 18:37:45.979907 938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f59e97528014856b2665408eb4c66a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:37:45.985121 938 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f59e97528014856b2665408eb4c66a0 [term 1 LEADER]: Becoming Leader. State: Replica: 0f59e97528014856b2665408eb4c66a0, State: Running, Role: LEADER
I20260620 18:37:45.985764 938 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f59e97528014856b2665408eb4c66a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f59e97528014856b2665408eb4c66a0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34293 } } peers { permanent_uuid: "18a834b0ead34344826f142564fe8858" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 35945 } } peers { permanent_uuid: "290918322a804cf3aa2d695379d92a29" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 44749 } }
I20260620 18:37:45.995726 949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f59e97528014856b2665408eb4c66a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f59e97528014856b2665408eb4c66a0. Latest consensus state: current_term: 1 leader_uuid: "0f59e97528014856b2665408eb4c66a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f59e97528014856b2665408eb4c66a0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34293 } } peers { permanent_uuid: "18a834b0ead34344826f142564fe8858" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 35945 } } peers { permanent_uuid: "290918322a804cf3aa2d695379d92a29" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 44749 } } }
I20260620 18:37:45.996173 949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f59e97528014856b2665408eb4c66a0 [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:45.998951 843 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.60" port: 44749 } attempt: 5
W20260620 18:37:46.004793 871 tablet.cc:2369] T 00000000000000000000000000000000 P 290918322a804cf3aa2d695379d92a29: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:37:46.009097 954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 20 18:37:46 dist-test-slave-cbn3 krb5kdc[689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980665, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.61@KRBTEST.COM for kudu/127.10.131.60@KRBTEST.COM
W20260620 18:37:46.030514 723 tablet.cc:2369] T 00000000000000000000000000000000 P 0f59e97528014856b2665408eb4c66a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:37:46.021320 954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:37:46.052657 846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.999774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:44749 (local address 127.0.0.1:54012)
0620 18:37:46.000130 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:46.000155 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:46.000307 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:46.001042 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:46.001050 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:46.001731 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:46.002028 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:46.002044 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:46.004105 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:46.004115 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:46.006077 (+ 1962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:46.006098 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:46.007632 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:46.007641 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:46.007991 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:46.008880 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:46.008915 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:46.012654 (+ 3739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:46.043043 (+ 30389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:46.043050 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:46.043054 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:46.043694 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:46.045411 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:46.045416 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:46.045419 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:46.045597 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:46.051573 (+ 5976us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:46.051579 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:46.051746 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 18:37:46.051906 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 18:37:46.052251 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198}
I20260620 18:37:46.054210 920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:45.999855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54012 (local address 127.10.131.60:44749)
0620 18:37:46.000090 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:46.000099 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:46.000448 (+ 349us) server_negotiation.cc:407] Connection header received
0620 18:37:46.000537 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:46.000546 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:46.000651 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:46.000762 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:46.002324 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:46.003885 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:46.006299 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:46.007194 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:46.007340 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:46.013005 (+ 5665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:46.013016 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:46.013023 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:46.013083 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:46.039313 (+ 26230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:46.044434 (+ 5121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:46.044441 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:46.044447 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:46.044639 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:46.045830 (+ 1191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:46.045839 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:46.045845 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:46.051419 (+ 5574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:46.053271 (+ 1852us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:46.053421 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 18:37:46.053697 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:37:46.060876 843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18a834b0ead34344826f142564fe8858: Bootstrap starting.
I20260620 18:37:46.064945 848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:46.044169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56354 (local address 127.10.131.61:35945)
0620 18:37:46.044333 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:46.044339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:46.046868 (+ 2529us) server_negotiation.cc:407] Connection header received
0620 18:37:46.047110 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:46.047115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:46.047184 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:46.047246 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:46.049700 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:46.050779 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:46.052989 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:46.053587 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:46.053684 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:46.057276 (+ 3592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:46.057287 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:46.057292 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:46.057344 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:46.061399 (+ 4055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:46.062288 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:46.062294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:46.062297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:46.062506 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:46.063041 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:46.063048 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:46.063051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:46.063476 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:46.063713 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:46.064308 (+ 595us) server_negotiation.cc:299] Negotiation successful
0620 18:37:46.064584 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:37:46.065693 843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18a834b0ead34344826f142564fe8858: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:46.067093 843 log.cc:824] T 00000000000000000000000000000000 P 18a834b0ead34344826f142564fe8858: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:46.026171 702 master_runner.cc:186] Error getting master registration for 127.10.131.61:35945: OK, instance_id { permanent_uuid: "18a834b0ead34344826f142564fe8858" instance_seqno: 1781980665274398 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:37:46.069896 702 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:37:46.084050 843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18a834b0ead34344826f142564fe8858: No bootstrap required, opened a new log
I20260620 18:37:46.093686 892 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 290918322a804cf3aa2d695379d92a29 [term 1 FOLLOWER]: Refusing update from remote peer 0f59e97528014856b2665408eb4c66a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:37:46.094926 949 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0f59e97528014856b2665408eb4c66a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "290918322a804cf3aa2d695379d92a29" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 44749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:37:46.099740 920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:46.061703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54026 (local address 127.10.131.60:44749)
0620 18:37:46.066013 (+ 4310us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:46.066021 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:46.069811 (+ 3790us) server_negotiation.cc:407] Connection header received
0620 18:37:46.069892 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:46.069898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:46.069985 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:46.070084 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:46.072350 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:46.073458 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:46.077063 (+ 3605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:46.077723 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:46.077849 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:46.085171 (+ 7322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:46.085182 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:46.085189 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:46.085231 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:46.090803 (+ 5572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:46.096576 (+ 5773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:46.096593 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:46.096596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:46.096764 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:46.097249 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:46.097254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:46.097258 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:46.097735 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:46.098792 (+ 1057us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:46.098941 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 18:37:46.099231 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4190,"thread_start_us":14977,"threads_started":1}
I20260620 18:37:46.108299 773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:46.042893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43802 (local address 127.10.131.62:34293)
0620 18:37:46.043987 (+ 1094us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:46.043993 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:46.061005 (+ 17012us) server_negotiation.cc:407] Connection header received
0620 18:37:46.078096 (+ 17091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:46.078105 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:46.078193 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:46.078267 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:46.083817 (+ 5550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:46.084954 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:46.091943 (+ 6989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:46.092692 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:46.092800 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:46.100597 (+ 7797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:46.100604 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:46.100609 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:46.100649 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:46.104527 (+ 3878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:46.105516 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:46.105522 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:46.105525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:46.105699 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:46.106173 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:46.106178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:46.106181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:46.106591 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:46.107024 (+ 433us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:46.107650 (+ 626us) server_negotiation.cc:299] Negotiation successful
0620 18:37:46.107914 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":931}
I20260620 18:37:46.117455 843 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18a834b0ead34344826f142564fe8858 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f59e97528014856b2665408eb4c66a0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34293 } } peers { permanent_uuid: "18a834b0ead34344826f142564fe8858" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 35945 } } peers { permanent_uuid: "290918322a804cf3aa2d695379d92a29" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 44749 } }
I20260620 18:37:46.117971 843 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18a834b0ead34344826f142564fe8858 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18a834b0ead34344826f142564fe8858, State: Initialized, Role: FOLLOWER
I20260620 18:37:46.118536 843 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18a834b0ead34344826f142564fe8858 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f59e97528014856b2665408eb4c66a0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34293 } } peers { permanent_uuid: "18a834b0ead34344826f142564fe8858" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 35945 } } peers { permanent_uuid: "290918322a804cf3aa2d695379d92a29" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 44749 } }
I20260620 18:37:46.123576 965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18a834b0ead34344826f142564fe8858 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f59e97528014856b2665408eb4c66a0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34293 } } peers { permanent_uuid: "18a834b0ead34344826f142564fe8858" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 35945 } } peers { permanent_uuid: "290918322a804cf3aa2d695379d92a29" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 44749 } } }
I20260620 18:37:46.124243 965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18a834b0ead34344826f142564fe8858 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:37:46.125687 924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 290918322a804cf3aa2d695379d92a29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f59e97528014856b2665408eb4c66a0. Latest consensus state: current_term: 1 leader_uuid: "0f59e97528014856b2665408eb4c66a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f59e97528014856b2665408eb4c66a0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34293 } } peers { permanent_uuid: "18a834b0ead34344826f142564fe8858" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 35945 } } peers { permanent_uuid: "290918322a804cf3aa2d695379d92a29" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 44749 } } }
I20260620 18:37:46.126009 924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 290918322a804cf3aa2d695379d92a29 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:37:46.129892 960 mvcc.cc:205] Tried to move back new op lower bound from 7298992808275828736 to 7298992807883837440. Current Snapshot: MvccSnapshot[applied={T|T < 7298992808275828736}]
I20260620 18:37:46.131558 843 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18a834b0ead34344826f142564fe8858 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:37:46.131985 817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18a834b0ead34344826f142564fe8858 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:46.135885 817 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 18a834b0ead34344826f142564fe8858 [term 1 FOLLOWER]: Refusing update from remote peer 0f59e97528014856b2665408eb4c66a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:37:46.136994 949 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0f59e97528014856b2665408eb4c66a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a834b0ead34344826f142564fe8858" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 35945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:37:46.138069 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f59e97528014856b2665408eb4c66a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "18a834b0ead34344826f142564fe8858"
I20260620 18:37:46.140913 818 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 18a834b0ead34344826f142564fe8858 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f59e97528014856b2665408eb4c66a0 in term 1.
I20260620 18:37:46.146616 949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f59e97528014856b2665408eb4c66a0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0f59e97528014856b2665408eb4c66a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f59e97528014856b2665408eb4c66a0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34293 } } peers { permanent_uuid: "18a834b0ead34344826f142564fe8858" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 35945 } } peers { permanent_uuid: "290918322a804cf3aa2d695379d92a29" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 44749 } } }
I20260620 18:37:46.146957 949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f59e97528014856b2665408eb4c66a0 [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:46.153403 924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 290918322a804cf3aa2d695379d92a29 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f59e97528014856b2665408eb4c66a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f59e97528014856b2665408eb4c66a0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34293 } } peers { permanent_uuid: "18a834b0ead34344826f142564fe8858" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 35945 } } peers { permanent_uuid: "290918322a804cf3aa2d695379d92a29" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 44749 } } }
I20260620 18:37:46.153693 924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 290918322a804cf3aa2d695379d92a29 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:37:46.159441 954 catalog_manager.cc:1347] Generated new cluster ID: 487904b1cfd941a99eb1f35b1e34593c
I20260620 18:37:46.159546 954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:37:46.197222 965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18a834b0ead34344826f142564fe8858 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f59e97528014856b2665408eb4c66a0. Latest consensus state: current_term: 1 leader_uuid: "0f59e97528014856b2665408eb4c66a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f59e97528014856b2665408eb4c66a0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34293 } } peers { permanent_uuid: "18a834b0ead34344826f142564fe8858" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 35945 } } peers { permanent_uuid: "290918322a804cf3aa2d695379d92a29" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 44749 } } }
I20260620 18:37:46.197593 965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18a834b0ead34344826f142564fe8858 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:37:46.199208 974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18a834b0ead34344826f142564fe8858 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f59e97528014856b2665408eb4c66a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f59e97528014856b2665408eb4c66a0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34293 } } peers { permanent_uuid: "18a834b0ead34344826f142564fe8858" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 35945 } } peers { permanent_uuid: "290918322a804cf3aa2d695379d92a29" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 44749 } } }
I20260620 18:37:46.199654 974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18a834b0ead34344826f142564fe8858 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:37:46.225260 938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f59e97528014856b2665408eb4c66a0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0f59e97528014856b2665408eb4c66a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f59e97528014856b2665408eb4c66a0" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 34293 } } peers { permanent_uuid: "18a834b0ead34344826f142564fe8858" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 35945 } } peers { permanent_uuid: "290918322a804cf3aa2d695379d92a29" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 44749 } } }
I20260620 18:37:46.225697 938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f59e97528014856b2665408eb4c66a0 [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:46.231427 954 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:37:46.233318 954 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:37:46.250764 982 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 18:37:46.259696 954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f59e97528014856b2665408eb4c66a0: Generated new TSK 0
I20260620 18:37:46.261776 954 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 18:37:46.356832 848 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:37:46.342545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56362 (local address 127.10.131.61:35945)
0620 18:37:46.342755 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:46.342761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:46.353976 (+ 11215us) server_negotiation.cc:407] Connection header received
0620 18:37:46.355258 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:46.355266 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:46.355363 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:46.355438 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:46.356569 (+ 1131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56362: BlockingRecv error: recv got EOF from 127.0.0.1:56362 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 18:37:46.357424 773 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:37:46.342577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43806 (local address 127.10.131.62:34293)
0620 18:37:46.342806 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:46.342812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:46.355426 (+ 12614us) server_negotiation.cc:407] Connection header received
0620 18:37:46.355857 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:46.355865 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:46.355954 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:46.356032 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:46.357175 (+ 1143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43806: BlockingRecv error: recv got EOF from 127.0.0.1:43806 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 18:37:46.359856 920 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:37:46.354501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54034 (local address 127.10.131.60:44749)
0620 18:37:46.354750 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:46.354759 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:46.358260 (+ 3501us) server_negotiation.cc:407] Connection header received
0620 18:37:46.358503 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:46.358510 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:46.358588 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:46.358659 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:46.359624 (+ 965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54034: BlockingRecv error: recv got EOF from 127.0.0.1:54034 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 18:37:46.368177 848 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:37:46.365994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56370 (local address 127.10.131.61:35945)
0620 18:37:46.366168 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:46.366174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:46.366643 (+ 469us) server_negotiation.cc:407] Connection header received
0620 18:37:46.366869 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:46.366875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:46.367003 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:46.367121 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:46.367894 (+ 773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56370: BlockingRecv error: recv got EOF from 127.0.0.1:56370 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 18:37:46.378868 848 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:37:46.374428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56374 (local address 127.10.131.61:35945)
0620 18:37:46.374600 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:46.374606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:46.377397 (+ 2791us) server_negotiation.cc:407] Connection header received
0620 18:37:46.377597 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:46.377602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:46.377696 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:46.377801 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:46.378616 (+ 815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56374: BlockingRecv error: recv got EOF from 127.0.0.1:56374 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 18:37:46.391158 848 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:37:46.383853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56376 (local address 127.10.131.61:35945)
0620 18:37:46.384026 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:46.384032 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:46.389531 (+ 5499us) server_negotiation.cc:407] Connection header received
0620 18:37:46.389748 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:46.389754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:46.389874 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:46.390000 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:46.390910 (+ 910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56376: BlockingRecv error: recv got EOF from 127.0.0.1:56376 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 18:37:46.405274 848 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:37:46.402977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56388 (local address 127.10.131.61:35945)
0620 18:37:46.403233 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:46.403241 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:46.403771 (+ 530us) server_negotiation.cc:407] Connection header received
0620 18:37:46.403975 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:46.403980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:46.404065 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:46.404166 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:46.405046 (+ 880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56388: BlockingRecv error: recv got EOF from 127.0.0.1:56388 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
W20260620 18:37:46.419293 848 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:37:46.417060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56394 (local address 127.10.131.61:35945)
0620 18:37:46.417253 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:46.417259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:46.417819 (+ 560us) server_negotiation.cc:407] Connection header received
0620 18:37:46.418026 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:46.418032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:46.418129 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:46.418247 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:46.419044 (+ 797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56394: BlockingRecv error: recv got EOF from 127.0.0.1:56394 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 18:37:46.441293 848 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:37:46.438963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56398 (local address 127.10.131.61:35945)
0620 18:37:46.439155 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:46.439161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:46.439704 (+ 543us) server_negotiation.cc:407] Connection header received
0620 18:37:46.439925 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:46.439931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:46.440019 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:46.440123 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:46.441036 (+ 913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56398: BlockingRecv error: recv got EOF from 127.0.0.1:56398 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 18:37:46.479151 848 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:37:46.477116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56402 (local address 127.10.131.61:35945)
0620 18:37:46.477345 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:46.477351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:46.477746 (+ 395us) server_negotiation.cc:407] Connection header received
0620 18:37:46.477903 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:46.477910 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:46.478008 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:46.478138 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:46.478867 (+ 729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56402: BlockingRecv error: recv got EOF from 127.0.0.1:56402 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260620 18:37:46.555692 848 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:37:46.553490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56416 (local address 127.10.131.61:35945)
0620 18:37:46.553707 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:46.553715 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:46.554139 (+ 424us) server_negotiation.cc:407] Connection header received
0620 18:37:46.554292 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:46.554299 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:46.554410 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:46.554560 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:46.555463 (+ 903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56416: BlockingRecv error: recv got EOF from 127.0.0.1:56416 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 18:37:46.691951 848 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:37:46.690243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56424 (local address 127.10.131.61:35945)
0620 18:37:46.690502 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:46.690509 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:46.690678 (+ 169us) server_negotiation.cc:407] Connection header received
0620 18:37:46.690837 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:46.690841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:46.690911 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:46.690998 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:46.691702 (+ 704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56424: BlockingRecv error: recv got EOF from 127.0.0.1:56424 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
I20260620 18:37:46.870910 935 catalog_manager.cc:1259] Loaded cluster ID: 487904b1cfd941a99eb1f35b1e34593c
I20260620 18:37:46.871006 935 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 290918322a804cf3aa2d695379d92a29: loading cluster ID for follower catalog manager: success
I20260620 18:37:46.873483 935 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 290918322a804cf3aa2d695379d92a29: acquiring CA information for follower catalog manager: success
I20260620 18:37:46.875154 935 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 290918322a804cf3aa2d695379d92a29: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 18:37:46.953305 848 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:37:46.951483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56440 (local address 127.10.131.61:35945)
0620 18:37:46.951674 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:46.951679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:46.951963 (+ 284us) server_negotiation.cc:407] Connection header received
0620 18:37:46.952131 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:46.952135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:46.952213 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:46.952308 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:46.953098 (+ 790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56440: BlockingRecv error: recv got EOF from 127.0.0.1:56440 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:37:47.254915 982 catalog_manager.cc:1259] Loaded cluster ID: 487904b1cfd941a99eb1f35b1e34593c
I20260620 18:37:47.255089 982 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 18a834b0ead34344826f142564fe8858: loading cluster ID for follower catalog manager: success
I20260620 18:37:47.257810 982 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 18a834b0ead34344826f142564fe8858: acquiring CA information for follower catalog manager: success
I20260620 18:37:47.259651 982 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 18a834b0ead34344826f142564fe8858: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:37:47.478552 1050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:47.472236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56450 (local address 127.10.131.61:35945)
0620 18:37:47.473123 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:47.473129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:47.473148 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:37:47.473218 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:47.473222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:47.473360 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:47.473492 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:37:47.473918 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:47.474757 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:47.476143 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:47.476621 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:47.476768 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:47.477371 (+ 603us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:37:47.477901 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:37:47.478070 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:47.478126 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 18:37:47.478210 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":527,"threads_started":1}
I20260620 18:37:47.492311 1056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:47.485009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54038 (local address 127.10.131.60:44749)
0620 18:37:47.485502 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:47.485509 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:47.485864 (+ 355us) server_negotiation.cc:407] Connection header received
0620 18:37:47.486074 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:47.486080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:47.486301 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:47.486490 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:37:47.487118 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:47.488188 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:47.489542 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:47.490204 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:47.490350 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:47.490812 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:37:47.491547 (+ 735us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:37:47.491719 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:47.491894 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 18:37:47.491984 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":241,"threads_started":1}
I20260620 18:37:47.509618 1050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:47.499310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56464 (local address 127.10.131.61:35945)
0620 18:37:47.499556 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:47.499562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:47.501628 (+ 2066us) server_negotiation.cc:407] Connection header received
0620 18:37:47.501894 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:47.501899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:47.501984 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:47.502053 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:37:47.503408 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:47.504653 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:47.506956 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:47.507716 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:47.507806 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:47.508773 (+ 967us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:37:47.508941 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:37:47.509000 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:47.509223 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 18:37:47.509321 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 18:37:47.509670 1056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:47.502068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54052 (local address 127.10.131.60:44749)
0620 18:37:47.502424 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:47.502431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:47.503012 (+ 581us) server_negotiation.cc:407] Connection header received
0620 18:37:47.503311 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:47.503316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:47.503413 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:47.503477 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:37:47.504200 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:47.505425 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:47.507588 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:47.508183 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:47.508269 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:47.508966 (+ 697us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:37:47.509085 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:37:47.509144 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:47.509352 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 18:37:47.509423 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221}
I20260620 18:37:47.509706 1063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:47.499720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43822 (local address 127.10.131.62:34293)
0620 18:37:47.501258 (+ 1538us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:47.501266 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:47.501290 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:37:47.501414 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:47.501420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:47.501640 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:47.502894 (+ 1254us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:37:47.503082 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:47.504183 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:47.506533 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:47.507179 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:47.507840 (+ 661us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:47.508039 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:37:47.508488 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:37:47.509255 (+ 767us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:47.509311 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 18:37:47.509397 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1407,"thread_start_us":1314,"threads_started":1}
I20260620 18:37:47.515491 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 702
I20260620 18:37:47.516736 764 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:47.632980 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 702
I20260620 18:37:47.653447 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 776
I20260620 18:37:47.654662 838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:47.760515 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 776
I20260620 18:37:47.783774 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 850
I20260620 18:37:47.784931 912 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:47.890208 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 850
2026-06-20T18:37:47Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4903 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:37:47 dist-test-slave-cbn3 krb5kdc[1070](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:37:47 dist-test-slave-cbn3 krb5kdc[1070](info): set up 2 sockets
Jun 20 18:37:47 dist-test-slave-cbn3 krb5kdc[1070](info): commencing operation
krb5kdc: starting...
W20260620 18:37:48.977486 10764 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/krb5kdc/test-user.keytab.
Jun 20 18:37:49 dist-test-slave-cbn3 krb5kdc[1070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:37:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:37:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.131.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
Entry for principal kudu/127.10.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab.
I20260620 18:37:49.229117 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:40351
--webserver_interface=127.10.131.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:34639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:40351,127.10.131.61:40401,127.10.131.60:41187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:37:49.551707 1083 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:37:49.552086 1083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:49.552212 1083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:49.552516 1083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:49.561498 1083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:37:49.561641 1083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:49.561745 1083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:49.561859 1083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:37:49.561949 1083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:37:49.571844 1083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:34639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:40351,127.10.131.61:40401,127.10.131.60:41187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.62.keytab
--principal=kudu/127.10.131.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.62:40351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.131.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:37:49.573181 1083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:49.574501 1083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:49.586406 1091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:49.586508 1088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:49.586467 1089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:49.587477 1083 server_base.cc:1034] running on GCE node
I20260620 18:37:49.588260 1083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:49.589668 1083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:49.590902 1083 hybrid_clock.cc:648] HybridClock initialized: now 1781980669590841 us; error 32 us; skew 500 ppm
I20260620 18:37:49.591238 1083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:49 dist-test-slave-cbn3 krb5kdc[1070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980669, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:49.594410 1083 init.cc:377] Logged in from keytab as kudu/127.10.131.62@KRBTEST.COM (short username kudu)
I20260620 18:37:49.595886 1083 webserver.cc:460] Webserver started at http://127.10.131.62:45969/ using document root <none> and password file <none>
I20260620 18:37:49.596556 1083 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:49.596674 1083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:49.597054 1083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:49.599972 1083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-0/data/instance:
uuid: "ca24be77ad6f4c158ce9c3504fe9a39e"
format_stamp: "Formatted at 2026-06-20 18:37:49 on dist-test-slave-cbn3"
server_key: "e57a12aaaae613b5a7286075250566f9"
server_key_iv: "83d4729349eb11a215620324ec04371b"
server_key_version: "encryptionkey@0"
I20260620 18:37:49.600749 1083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-0/wal/instance:
uuid: "ca24be77ad6f4c158ce9c3504fe9a39e"
format_stamp: "Formatted at 2026-06-20 18:37:49 on dist-test-slave-cbn3"
server_key: "e57a12aaaae613b5a7286075250566f9"
server_key_iv: "83d4729349eb11a215620324ec04371b"
server_key_version: "encryptionkey@0"
I20260620 18:37:49.605744 1083 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:37:49.609936 1098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:49.611694 1083 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 18:37:49.611881 1083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-0/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-0/wal
uuid: "ca24be77ad6f4c158ce9c3504fe9a39e"
format_stamp: "Formatted at 2026-06-20 18:37:49 on dist-test-slave-cbn3"
server_key: "e57a12aaaae613b5a7286075250566f9"
server_key_iv: "83d4729349eb11a215620324ec04371b"
server_key_version: "encryptionkey@0"
I20260620 18:37:49.612129 1083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:49.635355 1083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:49.639343 1083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:49.639611 1083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:49.657438 1083 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.62:40351
I20260620 18:37:49.657474 1149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.62:40351 every 8 connection(s)
I20260620 18:37:49.658766 1083 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-0/data/info.pb
I20260620 18:37:49.662292 1150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:49.663743 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 1083
I20260620 18:37:49.663946 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-0/wal/instance
I20260620 18:37:49.664508 10764 external_mini_cluster.cc:1428] Setting key cf50388080cc399f8d024a5f0f2f4cd3
I20260620 18:37:49.671728 1150 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.131.62" port: 40351 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:37:49 dist-test-slave-cbn3 krb5kdc[1070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980669, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.131.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.61@KRBTEST.COM" created.
I20260620 18:37:49.695930 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:49.675715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56962 (local address 127.10.131.62:40351)
0620 18:37:49.676753 (+ 1038us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:49.676759 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:49.676799 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:37:49.679078 (+ 2279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:49.679088 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:49.679402 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:49.679589 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:49.680812 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:49.682654 (+ 1842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:49.684117 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:49.685118 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:49.685297 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:49.688915 (+ 3618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:49.688924 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:49.688935 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:49.688964 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:49.691760 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:49.692616 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:49.692622 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:49.692624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:49.692775 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:49.693217 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:49.693221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:49.693223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:49.693785 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:49.693942 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:49.694528 (+ 586us) server_negotiation.cc:299] Negotiation successful
0620 18:37:49.695000 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":179,"server-negotiator.queue_time_us":855,"thread_start_us":659,"threads_started":1}
I20260620 18:37:49.695948 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:49.675714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:40351 (local address 127.0.0.1:56962)
0620 18:37:49.676547 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:49.676603 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:49.678676 (+ 2073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:49.679684 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:49.679689 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:49.680244 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:49.680588 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:49.680606 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:49.682821 (+ 2215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:49.682825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:49.683968 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:49.683981 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:49.685336 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:49.685341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:49.685467 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:49.686116 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:49.686136 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:49.688634 (+ 2498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:49.691904 (+ 3270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:49.691919 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:49.691934 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:49.692480 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:49.692934 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:49.692938 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:49.692940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:49.693080 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:49.693921 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:49.693927 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:49.694109 (+ 182us) client_negotiation.cc:769] Sending connection context
0620 18:37:49.694312 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 18:37:49.694687 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":350,"threads_started":1}
I20260620 18:37:49.699065 1150 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.131.61" port: 40401 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:37:49.700199 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:49.699825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:40401 (local address 127.0.0.1:59828)
0620 18:37:49.700006 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:49.700098 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.61:40401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260620 18:37:49.700559 1100 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.131.61:40401: connect: Connection refused (error 111)
I20260620 18:37:49.702050 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:49.701739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:40401 (local address 127.0.0.1:59836)
0620 18:37:49.701875 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:49.701948 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.61:40401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 18:37:49.702363 1150 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.61:40401: Network error: Client connection negotiation failed: client connection to 127.10.131.61:40401: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.61.keytab.
Entry for principal kudu/127.10.131.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.61.keytab.
I20260620 18:37:49.723985 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.61:40401
--webserver_interface=127.10.131.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.61.keytab
--principal=kudu/127.10.131.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:34639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:40351,127.10.131.61:40401,127.10.131.60:41187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:37:49.751705 1150 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.61" port: 40401 } attempt: 1
I20260620 18:37:49.752923 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:49.752434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:40401 (local address 127.0.0.1:59842)
0620 18:37:49.752617 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:49.752740 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.61:40401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 18:37:49.753897 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:49.753586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:40401 (local address 127.0.0.1:59856)
0620 18:37:49.753712 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:49.753773 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.61:40401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 18:37:49.754150 1150 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.61:40401: Network error: Client connection negotiation failed: client connection to 127.10.131.61:40401: connect: Connection refused (error 111)
I20260620 18:37:49.822396 1150 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.61" port: 40401 } attempt: 2
I20260620 18:37:49.823551 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:49.823047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:40401 (local address 127.0.0.1:59862)
0620 18:37:49.823322 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:49.823409 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.61:40401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 18:37:49.824569 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:49.824198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:40401 (local address 127.0.0.1:59864)
0620 18:37:49.824352 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:49.824428 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.61:40401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 18:37:49.824821 1150 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.61:40401: Network error: Client connection negotiation failed: client connection to 127.10.131.61:40401: connect: Connection refused (error 111)
I20260620 18:37:49.916100 1150 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.61" port: 40401 } attempt: 3
I20260620 18:37:49.917274 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:49.916870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:40401 (local address 127.0.0.1:59870)
0620 18:37:49.917055 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:49.917144 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.61:40401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 18:37:49.918279 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:49.917966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:40401 (local address 127.0.0.1:59884)
0620 18:37:49.918085 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:49.918156 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.61:40401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 18:37:49.918483 1150 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.61:40401: Network error: Client connection negotiation failed: client connection to 127.10.131.61:40401: connect: Connection refused (error 111)
W20260620 18:37:50.048234 1157 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:37:50.048560 1157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:50.048632 1157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:50.048715 1157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:50.056033 1157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:37:50.056136 1157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:50.056195 1157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:50.056290 1157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:37:50.056388 1157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:37:50.061729 1150 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.61" port: 40401 } attempt: 4
I20260620 18:37:50.062992 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.062447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:40401 (local address 127.0.0.1:59894)
0620 18:37:50.062756 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:50.062861 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.61:40401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260620 18:37:50.064008 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.063677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:40401 (local address 127.0.0.1:59908)
0620 18:37:50.063813 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:50.063890 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.61:40401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 18:37:50.064273 1150 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.61:40401: Network error: Client connection negotiation failed: client connection to 127.10.131.61:40401: connect: Connection refused (error 111)
I20260620 18:37:50.067355 1157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:34639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:40351,127.10.131.61:40401,127.10.131.60:41187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.61.keytab
--principal=kudu/127.10.131.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.61:40401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.131.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:37:50.068641 1157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:50.069998 1157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:50.081570 1165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:50.081602 1163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:50.081527 1162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:50.082896 1157 server_base.cc:1034] running on GCE node
I20260620 18:37:50.083567 1157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:50.084895 1157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:50.086124 1157 hybrid_clock.cc:648] HybridClock initialized: now 1781980670086058 us; error 37 us; skew 500 ppm
I20260620 18:37:50.086457 1157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:50 dist-test-slave-cbn3 krb5kdc[1070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980670, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:50.093021 1157 init.cc:377] Logged in from keytab as kudu/127.10.131.61@KRBTEST.COM (short username kudu)
I20260620 18:37:50.094486 1157 webserver.cc:460] Webserver started at http://127.10.131.61:42305/ using document root <none> and password file <none>
I20260620 18:37:50.095201 1157 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:50.095335 1157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:50.095650 1157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:50.098615 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-1/data/instance:
uuid: "e73bcd5c670a4629a2616cb500345400"
format_stamp: "Formatted at 2026-06-20 18:37:50 on dist-test-slave-cbn3"
server_key: "d80ddb65e11d0aab0f2f709701da92d1"
server_key_iv: "a95e58be1bf61237edae1ed49386b387"
server_key_version: "encryptionkey@0"
I20260620 18:37:50.099390 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-1/wal/instance:
uuid: "e73bcd5c670a4629a2616cb500345400"
format_stamp: "Formatted at 2026-06-20 18:37:50 on dist-test-slave-cbn3"
server_key: "d80ddb65e11d0aab0f2f709701da92d1"
server_key_iv: "a95e58be1bf61237edae1ed49386b387"
server_key_version: "encryptionkey@0"
I20260620 18:37:50.104429 1157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 18:37:50.107812 1172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:50.109202 1157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:37:50.109387 1157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-1/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-1/wal
uuid: "e73bcd5c670a4629a2616cb500345400"
format_stamp: "Formatted at 2026-06-20 18:37:50 on dist-test-slave-cbn3"
server_key: "d80ddb65e11d0aab0f2f709701da92d1"
server_key_iv: "a95e58be1bf61237edae1ed49386b387"
server_key_version: "encryptionkey@0"
I20260620 18:37:50.109614 1157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:50.134440 1157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:50.138350 1157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:50.138609 1157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:50.156672 1157 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.61:40401
I20260620 18:37:50.156695 1223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.61:40401 every 8 connection(s)
I20260620 18:37:50.158007 1157 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-1/data/info.pb
I20260620 18:37:50.161556 1224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:50.168346 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 1157
I20260620 18:37:50.167783 1224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.131.62" port: 40351 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:37:50.168545 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-1/wal/instance
I20260620 18:37:50.169013 10764 external_mini_cluster.cc:1428] Setting key f227f14fcb37208125055abd2bf0b8fb
Jun 20 18:37:50 dist-test-slave-cbn3 krb5kdc[1070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980670, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.61@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:37:50.188380 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.170430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56976 (local address 127.10.131.62:40351)
0620 18:37:50.170608 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:50.170613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:50.171979 (+ 1366us) server_negotiation.cc:407] Connection header received
0620 18:37:50.173955 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:50.173961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:50.174055 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:50.174166 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:50.175630 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.176329 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.178511 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.178929 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.179068 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:50.182598 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:50.182603 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:50.182606 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:50.182639 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:50.185347 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:50.186075 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:50.186085 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:50.186087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:50.186202 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:50.186636 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:50.186641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:50.186644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:50.186932 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:50.187092 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:50.187826 (+ 734us) server_negotiation.cc:299] Negotiation successful
0620 18:37:50.188050 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:37:50.189421 1226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.171039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:40351 (local address 127.0.0.1:56976)
0620 18:37:50.171800 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:50.171856 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:50.173672 (+ 1816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:50.174409 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:50.174418 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:50.174939 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:50.175456 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:50.175472 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.176508 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.176511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:50.178314 (+ 1803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:50.178327 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.179158 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.179166 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:50.179398 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:50.180122 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:50.180142 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:50.182375 (+ 2233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:50.185494 (+ 3119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:50.185503 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:50.185521 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:50.185941 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:50.186313 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:50.186327 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:50.186331 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:50.186516 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:50.187061 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:50.187070 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:50.187449 (+ 379us) client_negotiation.cc:769] Sending connection context
0620 18:37:50.187703 (+ 254us) client_negotiation.cc:240] Negotiation successful
0620 18:37:50.188203 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":307,"threads_started":1}
I20260620 18:37:50.191147 1224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.131.61" port: 40401 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.131.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.131.60@KRBTEST.COM" created.
Jun 20 18:37:50 dist-test-slave-cbn3 krb5kdc[1070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980670, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.61@KRBTEST.COM for kudu/127.10.131.61@KRBTEST.COM
I20260620 18:37:50.208267 1226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.192110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:40401 (local address 127.0.0.1:59910)
0620 18:37:50.192339 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:50.192357 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:50.192474 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:50.193454 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:50.193457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:50.193916 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:50.194058 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:50.194067 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.195902 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.195910 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:50.197540 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:50.197557 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.198444 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.198452 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:50.198676 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:50.199316 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:50.199350 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:50.201884 (+ 2534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:50.205740 (+ 3856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:50.205746 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:50.205750 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:50.206103 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:50.206511 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:50.206516 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:50.206520 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:50.206644 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:50.207257 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:50.207262 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:50.207427 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 18:37:50.207572 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 18:37:50.207872 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 18:37:50.208359 1228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.192217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59910 (local address 127.10.131.61:40401)
0620 18:37:50.193038 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:50.193043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:50.193070 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:37:50.193117 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:50.193122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:50.193313 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:50.193429 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:50.194248 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.195747 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.197701 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.198270 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.198380 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:50.202093 (+ 3713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:50.202105 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:50.202120 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:50.202153 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:50.205594 (+ 3441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:50.206253 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:50.206263 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:50.206271 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:50.206383 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:50.206757 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:50.206764 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:50.206768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:50.207111 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:50.207350 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:50.207722 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 18:37:50.207989 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":591,"thread_start_us":426,"threads_started":1}
I20260620 18:37:50.210883 1224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.131.60" port: 41187 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:37:50.212114 1226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.211664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:41187 (local address 127.0.0.1:60402)
0620 18:37:50.211843 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:50.211961 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:41187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260620 18:37:50.212399 1175 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.131.60:41187: connect: Connection refused (error 111)
I20260620 18:37:50.214154 1226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.213640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:41187 (local address 127.0.0.1:60404)
0620 18:37:50.213856 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:50.213989 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:41187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 18:37:50.214399 1224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.60:41187: Network error: Client connection negotiation failed: client connection to 127.10.131.60:41187: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.131.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.60.keytab.
Entry for principal kudu/127.10.131.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.60.keytab.
I20260620 18:37:50.227322 10764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0dhafo/build/asan/bin/kudu
/tmp/dist-test-task0dhafo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.60:41187
--webserver_interface=127.10.131.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.60.keytab
--principal=kudu/127.10.131.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.131.20:34639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.131.62:40351,127.10.131.61:40401,127.10.131.60:41187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:37:50.263742 1224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.60" port: 41187 } attempt: 1
I20260620 18:37:50.265035 1226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.264513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:41187 (local address 127.0.0.1:60416)
0620 18:37:50.264766 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:50.264891 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:41187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260620 18:37:50.266006 1226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.265665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:41187 (local address 127.0.0.1:60424)
0620 18:37:50.265840 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:50.265909 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:41187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 18:37:50.266201 1224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.60:41187: Network error: Client connection negotiation failed: client connection to 127.10.131.60:41187: connect: Connection refused (error 111)
I20260620 18:37:50.334430 1224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.60" port: 41187 } attempt: 2
I20260620 18:37:50.335630 1226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.335189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:41187 (local address 127.0.0.1:60440)
0620 18:37:50.335386 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:50.335482 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:41187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 18:37:50.336716 1226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.336341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:41187 (local address 127.0.0.1:60448)
0620 18:37:50.336498 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:50.336584 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:41187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 18:37:50.336982 1224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.60:41187: Network error: Client connection negotiation failed: client connection to 127.10.131.60:41187: connect: Connection refused (error 111)
I20260620 18:37:50.363534 1150 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.61" port: 40401 } attempt: 5
Jun 20 18:37:50 dist-test-slave-cbn3 krb5kdc[1070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980669, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for kudu/127.10.131.61@KRBTEST.COM
I20260620 18:37:50.378176 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.364294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:40401 (local address 127.0.0.1:59916)
0620 18:37:50.364498 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:50.364520 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:50.364630 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:50.365013 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:50.365016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:50.365368 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:50.365541 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:50.365549 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.366967 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.366971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:50.367838 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:50.367849 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.368955 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.368959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:50.369119 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:50.369735 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:50.369752 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:50.371856 (+ 2104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:50.375933 (+ 4077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:50.375937 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:50.375940 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:50.376278 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:50.376663 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:50.376666 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:50.376669 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:50.376804 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:50.377459 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:50.377462 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:50.377589 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 18:37:50.377692 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 18:37:50.377941 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 18:37:50.378377 1228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.364389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59916 (local address 127.10.131.61:40401)
0620 18:37:50.364594 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:50.364600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:50.364624 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:37:50.364750 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:50.364756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:50.364837 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:50.364982 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:50.365693 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.366830 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.367994 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.368783 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.368953 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:50.372043 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:50.372050 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:50.372054 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:50.372092 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:50.375799 (+ 3707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:50.376394 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:50.376399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:50.376403 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:50.376544 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:50.376965 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:50.376969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:50.376972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:50.377340 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:50.377471 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:50.377854 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 18:37:50.378074 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 18:37:50.379314 1150 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.131.60" port: 41187 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:37:50.380277 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.379941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:41187 (local address 127.0.0.1:60462)
0620 18:37:50.380089 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:50.380160 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:41187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 18:37:50.381156 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.380880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:41187 (local address 127.0.0.1:60474)
0620 18:37:50.381002 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:50.381061 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:41187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 18:37:50.381742 1150 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.60:41187: Network error: Client connection negotiation failed: client connection to 127.10.131.60:41187: connect: Connection refused (error 111)
I20260620 18:37:50.428216 1224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.60" port: 41187 } attempt: 3
I20260620 18:37:50.429476 1226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.428938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:41187 (local address 127.0.0.1:60478)
0620 18:37:50.429166 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:50.429275 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:41187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 18:37:50.430697 1226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.430100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:41187 (local address 127.0.0.1:60492)
0620 18:37:50.430377 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:50.430496 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:41187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
W20260620 18:37:50.430934 1224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.60:41187: Network error: Client connection negotiation failed: client connection to 127.10.131.60:41187: connect: Connection refused (error 111)
I20260620 18:37:50.432924 1150 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.60" port: 41187 } attempt: 1
I20260620 18:37:50.433930 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.433534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:41187 (local address 127.0.0.1:60498)
0620 18:37:50.433698 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:50.433774 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:41187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 18:37:50.435024 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.434595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:41187 (local address 127.0.0.1:60510)
0620 18:37:50.434767 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:50.434862 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:41187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 18:37:50.435256 1150 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.60:41187: Network error: Client connection negotiation failed: client connection to 127.10.131.60:41187: connect: Connection refused (error 111)
I20260620 18:37:50.503491 1150 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.60" port: 41187 } attempt: 2
I20260620 18:37:50.504979 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.504152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:41187 (local address 127.0.0.1:60526)
0620 18:37:50.504425 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:50.504575 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:41187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 18:37:50.506138 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.505730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:41187 (local address 127.0.0.1:60536)
0620 18:37:50.505903 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:50.505996 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:41187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 18:37:50.506376 1150 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.60:41187: Network error: Client connection negotiation failed: client connection to 127.10.131.60:41187: connect: Connection refused (error 111)
W20260620 18:37:50.559445 1231 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:37:50.559808 1231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:37:50.559926 1231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:37:50.560045 1231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:37:50.567639 1231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:37:50.567731 1231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:37:50.567857 1231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:37:50.567931 1231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:37:50.568019 1231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:37:50.574146 1224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.60" port: 41187 } attempt: 4
I20260620 18:37:50.575294 1226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.574773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:41187 (local address 127.0.0.1:60552)
0620 18:37:50.575010 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:50.575110 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:41187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 18:37:50.576301 1226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.575950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:41187 (local address 127.0.0.1:60554)
0620 18:37:50.576090 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:50.576178 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:41187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 18:37:50.576499 1224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.60:41187: Network error: Client connection negotiation failed: client connection to 127.10.131.60:41187: connect: Connection refused (error 111)
I20260620 18:37:50.578748 1231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.131.20:34639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.131.62:40351,127.10.131.61:40401,127.10.131.60:41187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/krb5kdc/kudu_127.10.131.60.keytab
--principal=kudu/127.10.131.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.131.60:41187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.131.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:37:50.579967 1231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:37:50.581388 1231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:37:50.593118 1239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:50.593174 1237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:37:50.593210 1236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:37:50.594321 1231 server_base.cc:1034] running on GCE node
I20260620 18:37:50.595084 1231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:37:50.596416 1231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:37:50.597659 1231 hybrid_clock.cc:648] HybridClock initialized: now 1781980670597585 us; error 48 us; skew 500 ppm
I20260620 18:37:50.597995 1231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:37:50 dist-test-slave-cbn3 krb5kdc[1070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980670, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:37:50.601246 1231 init.cc:377] Logged in from keytab as kudu/127.10.131.60@KRBTEST.COM (short username kudu)
I20260620 18:37:50.602632 1231 webserver.cc:460] Webserver started at http://127.10.131.60:34517/ using document root <none> and password file <none>
I20260620 18:37:50.603343 1231 fs_manager.cc:359] Metadata directory not provided
I20260620 18:37:50.603471 1231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:37:50.603787 1231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:37:50.606750 1231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-2/data/instance:
uuid: "b12a2d40d57a46939f49402474da9420"
format_stamp: "Formatted at 2026-06-20 18:37:50 on dist-test-slave-cbn3"
server_key: "34f48e45cf43d68f5fda15d48faa0aac"
server_key_iv: "ac01dfb5d7e4b80865bdb43e090c408d"
server_key_version: "encryptionkey@0"
I20260620 18:37:50.607666 1231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-2/wal/instance:
uuid: "b12a2d40d57a46939f49402474da9420"
format_stamp: "Formatted at 2026-06-20 18:37:50 on dist-test-slave-cbn3"
server_key: "34f48e45cf43d68f5fda15d48faa0aac"
server_key_iv: "ac01dfb5d7e4b80865bdb43e090c408d"
server_key_version: "encryptionkey@0"
I20260620 18:37:50.612588 1150 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.60" port: 41187 } attempt: 3
I20260620 18:37:50.613196 1231 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 18:37:50.613812 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.613289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:41187 (local address 127.0.0.1:60566)
0620 18:37:50.613516 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:50.613608 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:41187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 18:37:50.614923 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.614524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:41187 (local address 127.0.0.1:60576)
0620 18:37:50.614735 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:50.614810 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.131.60:41187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260620 18:37:50.615175 1150 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.131.60:41187: Network error: Client connection negotiation failed: client connection to 127.10.131.60:41187: connect: Connection refused (error 111)
I20260620 18:37:50.617307 1246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:37:50.618609 1231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:37:50.618794 1231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-2/data,/tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-2/wal
uuid: "b12a2d40d57a46939f49402474da9420"
format_stamp: "Formatted at 2026-06-20 18:37:50 on dist-test-slave-cbn3"
server_key: "34f48e45cf43d68f5fda15d48faa0aac"
server_key_iv: "ac01dfb5d7e4b80865bdb43e090c408d"
server_key_version: "encryptionkey@0"
I20260620 18:37:50.619019 1231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:37:50.632622 1231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:37:50.637058 1231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:37:50.637317 1231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:37:50.656548 1231 rpc_server.cc:307] RPC server started. Bound to: 127.10.131.60:41187
I20260620 18:37:50.656566 1297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.131.60:41187 every 8 connection(s)
I20260620 18:37:50.658417 1231 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-2/data/info.pb
I20260620 18:37:50.661409 10764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0dhafo/build/asan/bin/kudu as pid 1231
I20260620 18:37:50.661594 10764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0dhafo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781980314095189-10764-0/minicluster-data/master-2/wal/instance
I20260620 18:37:50.662177 1298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:37:50.662214 10764 external_mini_cluster.cc:1428] Setting key 1edea46fe569fca575f03ffea5802086
I20260620 18:37:50.671993 1298 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.131.62" port: 40351 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:37:50 dist-test-slave-cbn3 krb5kdc[1070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.60@KRBTEST.COM
Jun 20 18:37:50 dist-test-slave-cbn3 krb5kdc[1070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980670, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.60@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
I20260620 18:37:50.685809 1301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.664149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60586 (local address 127.10.131.60:41187)
0620 18:37:50.665011 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:50.665027 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:50.665080 (+ 53us) server_negotiation.cc:407] Connection header received
0620 18:37:50.667061 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:50.667087 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:50.667597 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:50.668026 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:50.669545 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.671425 (+ 1880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.672938 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.673688 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.673887 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:50.678571 (+ 4684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:50.678578 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:50.678588 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:50.678619 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:50.681775 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:50.682443 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:50.682451 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:50.682458 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:50.682646 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:50.683100 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:50.683105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:50.683107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:50.683714 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:50.683968 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:50.684439 (+ 471us) server_negotiation.cc:299] Negotiation successful
0620 18:37:50.684835 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":337,"threads_started":1}
I20260620 18:37:50.688633 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.674351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56986 (local address 127.10.131.62:40351)
0620 18:37:50.674627 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:50.674635 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:50.675267 (+ 632us) server_negotiation.cc:407] Connection header received
0620 18:37:50.675436 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:50.675444 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:50.675538 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:50.675668 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:50.676658 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.677507 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.679213 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.679616 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.679765 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:50.682991 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:50.682998 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:50.683002 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:50.683047 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:50.685954 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:50.686647 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:50.686653 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:50.686657 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:50.686790 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:50.687138 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:50.687143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:50.687147 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:50.687440 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:50.687603 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:50.688018 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 18:37:50.688234 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260620 18:37:50.688959 1302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.674348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.62:40351 (local address 127.0.0.1:56986)
0620 18:37:50.675120 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:50.675164 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:50.675297 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:50.675700 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:50.675705 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:50.676217 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:50.676444 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:50.676452 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.677658 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.677663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:50.679061 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:50.679073 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.679836 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.679844 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:50.680023 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:50.680567 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:50.680593 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:50.682823 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:50.686101 (+ 3278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:50.686111 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:50.686118 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:50.686528 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:50.686908 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:50.686912 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:50.686914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:50.687028 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:50.687565 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:50.687574 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:50.687747 (+ 173us) client_negotiation.cc:769] Sending connection context
0620 18:37:50.688244 (+ 497us) client_negotiation.cc:240] Negotiation successful
0620 18:37:50.688533 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":409,"threads_started":1}
I20260620 18:37:50.690462 1298 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.131.61" port: 40401 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:37:50 dist-test-slave-cbn3 krb5kdc[1070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.62@KRBTEST.COM
Jun 20 18:37:50 dist-test-slave-cbn3 krb5kdc[1070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980670, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.60@KRBTEST.COM for kudu/127.10.131.61@KRBTEST.COM
I20260620 18:37:50.704041 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.688900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56994 (local address 127.10.131.62:40351)
0620 18:37:50.689455 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:50.689458 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:50.689471 (+ 13us) server_negotiation.cc:407] Connection header received
0620 18:37:50.689520 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:50.689523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:50.689571 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:50.689686 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:50.691160 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.691923 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.693494 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.694108 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.694229 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:50.698635 (+ 4406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:50.698640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:50.698643 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:50.698672 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:50.701421 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:50.702106 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:50.702111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:50.702114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:50.702269 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:50.702652 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:50.702657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:50.702660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:50.702940 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:50.703215 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:50.703517 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 18:37:50.703727 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":955,"threads_started":1}
I20260620 18:37:50.704945 1302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.691115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.61:40401 (local address 127.0.0.1:59932)
0620 18:37:50.691248 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:50.691262 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:50.691329 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:50.692555 (+ 1226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:50.692559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:50.692993 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:50.693172 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:50.693184 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.694474 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.694482 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:50.695954 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:50.695975 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.696688 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.696694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:50.696917 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:50.697498 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:50.697514 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:50.699515 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:50.702664 (+ 3149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:50.702671 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:50.702675 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:50.703031 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:50.703409 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:50.703414 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:50.703418 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:50.703556 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:50.704081 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:50.704085 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:50.704255 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 18:37:50.704409 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 18:37:50.704657 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 18:37:50.705201 1228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.692125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59932 (local address 127.10.131.61:40401)
0620 18:37:50.692294 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:50.692298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:50.692313 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:37:50.692358 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:50.692362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:50.692416 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:50.692491 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:50.693318 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.694321 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.696121 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.696537 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.696633 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:50.699692 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:50.699697 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:50.699700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:50.699730 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:50.702544 (+ 2814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:50.703153 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:50.703158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:50.703162 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:50.703290 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:50.703661 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:50.703667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:50.703671 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:50.703980 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:50.704114 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:50.704496 (+ 382us) server_negotiation.cc:299] Negotiation successful
0620 18:37:50.704747 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:37:50.705966 1298 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.131.60" port: 41187 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:37:50 dist-test-slave-cbn3 krb5kdc[1070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980670, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.60@KRBTEST.COM for kudu/127.10.131.60@KRBTEST.COM
Jun 20 18:37:50 dist-test-slave-cbn3 krb5kdc[1070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.131.61@KRBTEST.COM
I20260620 18:37:50.720808 1302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.706657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:41187 (local address 127.0.0.1:60594)
0620 18:37:50.706801 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:50.706819 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:50.706914 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:50.707181 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:50.707187 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:50.707685 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:50.707856 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:50.707867 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.709260 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.709265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:50.710659 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:50.710672 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.711639 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.711644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:50.711850 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:50.712529 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:50.712555 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:50.715039 (+ 2484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:50.718522 (+ 3483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:50.718528 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:50.718532 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:50.718873 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:50.719221 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:50.719228 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:50.719231 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:50.719356 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:50.719822 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:50.719826 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:50.719965 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 18:37:50.720095 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 18:37:50.720405 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 18:37:50.720889 1301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.706789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60594 (local address 127.10.131.60:41187)
0620 18:37:50.706947 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:50.706951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:50.706964 (+ 13us) server_negotiation.cc:407] Connection header received
0620 18:37:50.706998 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:50.707001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:50.707050 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:50.707118 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:50.708014 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.709106 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.710790 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.711471 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.711594 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:50.715213 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:50.715218 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:50.715222 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:50.715248 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:50.718378 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:50.718982 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:50.718990 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:50.718994 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:50.719126 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:50.719451 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:50.719458 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:50.719461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:50.719715 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:50.719864 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:50.720171 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 18:37:50.720557 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":157,"server-negotiator.queue_time_us":62}
I20260620 18:37:50.721805 1228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.705812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59942 (local address 127.10.131.61:40401)
0620 18:37:50.705961 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:50.705965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:50.706181 (+ 216us) server_negotiation.cc:407] Connection header received
0620 18:37:50.706281 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:50.706284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:50.706346 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:50.706445 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:50.708025 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.709135 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.710790 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.711471 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.711594 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:50.716462 (+ 4868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:50.716466 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:50.716470 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:50.716511 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:50.718943 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:50.719552 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:50.719560 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:50.719565 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:50.719693 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:50.720059 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:50.720064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:50.720067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:50.720298 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:50.720567 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:50.720873 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 18:37:50.721511 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 18:37:50.726264 10764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:37:50.726802 1298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b12a2d40d57a46939f49402474da9420: Bootstrap starting.
I20260620 18:37:50.730187 1298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b12a2d40d57a46939f49402474da9420: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:50.731191 1298 log.cc:824] T 00000000000000000000000000000000 P b12a2d40d57a46939f49402474da9420: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:50.728665 1125 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ca24be77ad6f4c158ce9c3504fe9a39e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56994
I20260620 18:37:50.734149 1298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b12a2d40d57a46939f49402474da9420: No bootstrap required, opened a new log
I20260620 18:37:50.741045 1298 raft_consensus.cc:348] T 00000000000000000000000000000000 P b12a2d40d57a46939f49402474da9420 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca24be77ad6f4c158ce9c3504fe9a39e" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 40351 } } peers { permanent_uuid: "e73bcd5c670a4629a2616cb500345400" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 40401 } } peers { permanent_uuid: "b12a2d40d57a46939f49402474da9420" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 41187 } }
I20260620 18:37:50.741402 1298 raft_consensus.cc:729] T 00000000000000000000000000000000 P b12a2d40d57a46939f49402474da9420 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b12a2d40d57a46939f49402474da9420, State: Initialized, Role: FOLLOWER
I20260620 18:37:50.741852 1298 consensus_queue.cc:260] T 00000000000000000000000000000000 P b12a2d40d57a46939f49402474da9420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca24be77ad6f4c158ce9c3504fe9a39e" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 40351 } } peers { permanent_uuid: "e73bcd5c670a4629a2616cb500345400" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 40401 } } peers { permanent_uuid: "b12a2d40d57a46939f49402474da9420" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 41187 } }
I20260620 18:37:50.742705 1305 sys_catalog.cc:455] T 00000000000000000000000000000000 P b12a2d40d57a46939f49402474da9420 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca24be77ad6f4c158ce9c3504fe9a39e" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 40351 } } peers { permanent_uuid: "e73bcd5c670a4629a2616cb500345400" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 40401 } } peers { permanent_uuid: "b12a2d40d57a46939f49402474da9420" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 41187 } } }
I20260620 18:37:50.743012 1305 sys_catalog.cc:458] T 00000000000000000000000000000000 P b12a2d40d57a46939f49402474da9420 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:37:50.744005 1298 sys_catalog.cc:564] T 00000000000000000000000000000000 P b12a2d40d57a46939f49402474da9420 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 18:37:50.753083 1316 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b12a2d40d57a46939f49402474da9420: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:37:50.753227 1316 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:37:50.751138 1231 master_runner.cc:186] Error getting master registration for 127.10.131.62:40351: OK, instance_id { permanent_uuid: "ca24be77ad6f4c158ce9c3504fe9a39e" instance_seqno: 1781980669652270 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:37:50.758883 1231 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:37:50.792483 1150 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.60" port: 41187 } attempt: 4
Jun 20 18:37:50 dist-test-slave-cbn3 krb5kdc[1070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980669, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.62@KRBTEST.COM for kudu/127.10.131.60@KRBTEST.COM
I20260620 18:37:50.807381 1301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.793308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60596 (local address 127.10.131.60:41187)
0620 18:37:50.793528 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:50.793535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:50.793610 (+ 75us) server_negotiation.cc:407] Connection header received
0620 18:37:50.793815 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:50.793821 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:50.793918 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:50.794012 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:50.794762 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.795521 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.796783 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.797332 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.797457 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:50.801379 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:50.801385 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:50.801390 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:50.801430 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:50.804403 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:50.805222 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:50.805228 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:50.805235 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:50.805379 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:50.805810 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:50.805815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:50.805818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:50.806205 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:50.806360 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:50.806779 (+ 419us) server_negotiation.cc:299] Negotiation successful
0620 18:37:50.807009 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:37:50.807404 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.793277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:41187 (local address 127.0.0.1:60596)
0620 18:37:50.793497 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:50.793517 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:50.793671 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:50.794053 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:50.794059 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:50.794470 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:50.794619 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:50.794628 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.795663 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.795669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:50.796638 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:50.796648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.797530 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.797536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:50.797759 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:50.798385 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:50.798411 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:50.801190 (+ 2779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:50.804522 (+ 3332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:50.804527 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:50.804532 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:50.805111 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:50.805508 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:50.805514 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:50.805517 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:50.805701 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:50.806343 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:50.806349 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:50.806517 (+ 168us) client_negotiation.cc:769] Sending connection context
0620 18:37:50.806678 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 18:37:50.806971 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 18:37:50.813140 1150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca24be77ad6f4c158ce9c3504fe9a39e: Bootstrap starting.
I20260620 18:37:50.817098 1150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca24be77ad6f4c158ce9c3504fe9a39e: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:50.818099 1150 log.cc:824] T 00000000000000000000000000000000 P ca24be77ad6f4c158ce9c3504fe9a39e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:50.820708 1150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca24be77ad6f4c158ce9c3504fe9a39e: No bootstrap required, opened a new log
I20260620 18:37:50.827630 1150 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca24be77ad6f4c158ce9c3504fe9a39e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca24be77ad6f4c158ce9c3504fe9a39e" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 40351 } } peers { permanent_uuid: "e73bcd5c670a4629a2616cb500345400" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 40401 } } peers { permanent_uuid: "b12a2d40d57a46939f49402474da9420" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 41187 } }
I20260620 18:37:50.828033 1150 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca24be77ad6f4c158ce9c3504fe9a39e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca24be77ad6f4c158ce9c3504fe9a39e, State: Initialized, Role: FOLLOWER
I20260620 18:37:50.828473 1150 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca24be77ad6f4c158ce9c3504fe9a39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca24be77ad6f4c158ce9c3504fe9a39e" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 40351 } } peers { permanent_uuid: "e73bcd5c670a4629a2616cb500345400" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 40401 } } peers { permanent_uuid: "b12a2d40d57a46939f49402474da9420" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 41187 } }
I20260620 18:37:50.829360 1319 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca24be77ad6f4c158ce9c3504fe9a39e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca24be77ad6f4c158ce9c3504fe9a39e" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 40351 } } peers { permanent_uuid: "e73bcd5c670a4629a2616cb500345400" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 40401 } } peers { permanent_uuid: "b12a2d40d57a46939f49402474da9420" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 41187 } } }
I20260620 18:37:50.829747 1319 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca24be77ad6f4c158ce9c3504fe9a39e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:37:50.830408 1150 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca24be77ad6f4c158ce9c3504fe9a39e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:37:50.830549 1125 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca24be77ad6f4c158ce9c3504fe9a39e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 18:37:50.830736 1125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca24be77ad6f4c158ce9c3504fe9a39e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:50.834568 1125 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca24be77ad6f4c158ce9c3504fe9a39e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca24be77ad6f4c158ce9c3504fe9a39e" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 40351 } } peers { permanent_uuid: "e73bcd5c670a4629a2616cb500345400" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 40401 } } peers { permanent_uuid: "b12a2d40d57a46939f49402474da9420" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 41187 } }
I20260620 18:37:50.836800 1273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca24be77ad6f4c158ce9c3504fe9a39e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b12a2d40d57a46939f49402474da9420"
I20260620 18:37:50.837265 1273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b12a2d40d57a46939f49402474da9420 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:50.840615 1273 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b12a2d40d57a46939f49402474da9420 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca24be77ad6f4c158ce9c3504fe9a39e in term 1.
I20260620 18:37:50.841020 1125 leader_election.cc:288] T 00000000000000000000000000000000 P ca24be77ad6f4c158ce9c3504fe9a39e [CANDIDATE]: Term 1 election: Requested vote from peers e73bcd5c670a4629a2616cb500345400 (127.10.131.61:40401), b12a2d40d57a46939f49402474da9420 (127.10.131.60:41187)
I20260620 18:37:50.841276 1101 leader_election.cc:302] T 00000000000000000000000000000000 P ca24be77ad6f4c158ce9c3504fe9a39e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b12a2d40d57a46939f49402474da9420, ca24be77ad6f4c158ce9c3504fe9a39e; no voters:
I20260620 18:37:50.841909 1319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca24be77ad6f4c158ce9c3504fe9a39e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:37:50.852622 1319 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca24be77ad6f4c158ce9c3504fe9a39e [term 1 LEADER]: Becoming Leader. State: Replica: ca24be77ad6f4c158ce9c3504fe9a39e, State: Running, Role: LEADER
I20260620 18:37:50.853982 1319 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca24be77ad6f4c158ce9c3504fe9a39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca24be77ad6f4c158ce9c3504fe9a39e" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 40351 } } peers { permanent_uuid: "e73bcd5c670a4629a2616cb500345400" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 40401 } } peers { permanent_uuid: "b12a2d40d57a46939f49402474da9420" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 41187 } }
I20260620 18:37:50.865564 1327 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca24be77ad6f4c158ce9c3504fe9a39e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca24be77ad6f4c158ce9c3504fe9a39e. Latest consensus state: current_term: 1 leader_uuid: "ca24be77ad6f4c158ce9c3504fe9a39e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca24be77ad6f4c158ce9c3504fe9a39e" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 40351 } } peers { permanent_uuid: "e73bcd5c670a4629a2616cb500345400" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 40401 } } peers { permanent_uuid: "b12a2d40d57a46939f49402474da9420" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 41187 } } }
I20260620 18:37:50.865986 1327 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca24be77ad6f4c158ce9c3504fe9a39e [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:50.866874 1336 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:37:50.875677 1224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.131.60" port: 41187 } attempt: 5
I20260620 18:37:50.879415 1336 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 18:37:50 dist-test-slave-cbn3 krb5kdc[1070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781980670, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.131.61@KRBTEST.COM for kudu/127.10.131.60@KRBTEST.COM
W20260620 18:37:50.889288 1252 tablet.cc:2369] T 00000000000000000000000000000000 P b12a2d40d57a46939f49402474da9420: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:37:50.892141 1104 tablet.cc:2369] T 00000000000000000000000000000000 P ca24be77ad6f4c158ce9c3504fe9a39e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:37:50.914412 1301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.877022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60606 (local address 127.10.131.60:41187)
0620 18:37:50.877215 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:50.877220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:50.877238 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:37:50.877297 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:50.877302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:50.877378 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:50.877476 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:50.878623 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.879781 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.881622 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.882322 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.882445 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:50.891495 (+ 9050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:50.891505 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:50.891512 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:50.891553 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:50.896392 (+ 4839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:50.898707 (+ 2315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:50.898714 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:50.898755 (+ 41us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:50.898982 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:50.900129 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:50.900136 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:50.900139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:50.900597 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:50.900753 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:50.913776 (+ 13023us) server_negotiation.cc:299] Negotiation successful
0620 18:37:50.914049 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:37:50.916285 1226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.876196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.131.60:41187 (local address 127.0.0.1:60606)
0620 18:37:50.876532 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0620 18:37:50.876558 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 18:37:50.876688 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:37:50.877693 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:37:50.877698 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:37:50.878197 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:37:50.878402 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:50.878413 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.879972 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.879977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:50.881429 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:37:50.881443 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.882675 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.882681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:37:50.882934 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:50.883633 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:37:50.883792 (+ 159us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:37:50.891070 (+ 7278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:37:50.896832 (+ 5762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:50.896837 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:50.896867 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:50.898366 (+ 1499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:50.899342 (+ 976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:37:50.899349 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:37:50.899353 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:37:50.899888 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:37:50.912295 (+ 12407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:37:50.912301 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:37:50.912531 (+ 230us) client_negotiation.cc:769] Sending connection context
0620 18:37:50.912715 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 18:37:50.913125 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213}
I20260620 18:37:50.922674 1224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e73bcd5c670a4629a2616cb500345400: Bootstrap starting.
I20260620 18:37:50.926959 1224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e73bcd5c670a4629a2616cb500345400: Neither blocks nor log segments found. Creating new log.
I20260620 18:37:50.928350 1224 log.cc:824] T 00000000000000000000000000000000 P e73bcd5c670a4629a2616cb500345400: Log is configured to *not* fsync() on all Append() calls
I20260620 18:37:50.935304 1224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e73bcd5c670a4629a2616cb500345400: No bootstrap required, opened a new log
I20260620 18:37:50.871995 1083 master_runner.cc:186] Error getting master registration for 127.10.131.61:40401: OK, instance_id { permanent_uuid: "e73bcd5c670a4629a2616cb500345400" instance_seqno: 1781980670151570 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:37:50.943281 1083 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:37:50.946627 1273 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b12a2d40d57a46939f49402474da9420 [term 1 FOLLOWER]: Refusing update from remote peer ca24be77ad6f4c158ce9c3504fe9a39e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:37:50.947783 1327 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ca24be77ad6f4c158ce9c3504fe9a39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b12a2d40d57a46939f49402474da9420" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 41187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:37:50.962347 1224 raft_consensus.cc:348] T 00000000000000000000000000000000 P e73bcd5c670a4629a2616cb500345400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca24be77ad6f4c158ce9c3504fe9a39e" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 40351 } } peers { permanent_uuid: "e73bcd5c670a4629a2616cb500345400" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 40401 } } peers { permanent_uuid: "b12a2d40d57a46939f49402474da9420" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 41187 } }
I20260620 18:37:50.965560 1224 raft_consensus.cc:729] T 00000000000000000000000000000000 P e73bcd5c670a4629a2616cb500345400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e73bcd5c670a4629a2616cb500345400, State: Initialized, Role: FOLLOWER
I20260620 18:37:50.966076 1224 consensus_queue.cc:260] T 00000000000000000000000000000000 P e73bcd5c670a4629a2616cb500345400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca24be77ad6f4c158ce9c3504fe9a39e" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 40351 } } peers { permanent_uuid: "e73bcd5c670a4629a2616cb500345400" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 40401 } } peers { permanent_uuid: "b12a2d40d57a46939f49402474da9420" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 41187 } }
I20260620 18:37:50.967257 1345 sys_catalog.cc:455] T 00000000000000000000000000000000 P e73bcd5c670a4629a2616cb500345400 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca24be77ad6f4c158ce9c3504fe9a39e" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 40351 } } peers { permanent_uuid: "e73bcd5c670a4629a2616cb500345400" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 40401 } } peers { permanent_uuid: "b12a2d40d57a46939f49402474da9420" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 41187 } } }
I20260620 18:37:50.967916 1345 sys_catalog.cc:458] T 00000000000000000000000000000000 P e73bcd5c670a4629a2616cb500345400 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:37:50.972272 1224 sys_catalog.cc:564] T 00000000000000000000000000000000 P e73bcd5c670a4629a2616cb500345400 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:37:50.972337 1199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca24be77ad6f4c158ce9c3504fe9a39e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e73bcd5c670a4629a2616cb500345400"
I20260620 18:37:50.972687 1198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e73bcd5c670a4629a2616cb500345400 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:37:50.973992 1305 sys_catalog.cc:455] T 00000000000000000000000000000000 P b12a2d40d57a46939f49402474da9420 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca24be77ad6f4c158ce9c3504fe9a39e. Latest consensus state: current_term: 1 leader_uuid: "ca24be77ad6f4c158ce9c3504fe9a39e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca24be77ad6f4c158ce9c3504fe9a39e" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 40351 } } peers { permanent_uuid: "e73bcd5c670a4629a2616cb500345400" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 40401 } } peers { permanent_uuid: "b12a2d40d57a46939f49402474da9420" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 41187 } } }
I20260620 18:37:50.974319 1305 sys_catalog.cc:458] T 00000000000000000000000000000000 P b12a2d40d57a46939f49402474da9420 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:37:50.976256 1198 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e73bcd5c670a4629a2616cb500345400 [term 1 FOLLOWER]: Refusing update from remote peer ca24be77ad6f4c158ce9c3504fe9a39e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:37:50.980057 1199 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e73bcd5c670a4629a2616cb500345400 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca24be77ad6f4c158ce9c3504fe9a39e in term 1.
I20260620 18:37:50.980697 1343 mvcc.cc:205] Tried to move back new op lower bound from 7298992828184358912 to 7298992827829968896. Current Snapshot: MvccSnapshot[applied={T|T < 7298992828184358912}]
I20260620 18:37:50.980655 1319 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ca24be77ad6f4c158ce9c3504fe9a39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e73bcd5c670a4629a2616cb500345400" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 40401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:37:50.992616 1305 sys_catalog.cc:455] T 00000000000000000000000000000000 P b12a2d40d57a46939f49402474da9420 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ca24be77ad6f4c158ce9c3504fe9a39e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca24be77ad6f4c158ce9c3504fe9a39e" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 40351 } } peers { permanent_uuid: "e73bcd5c670a4629a2616cb500345400" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 40401 } } peers { permanent_uuid: "b12a2d40d57a46939f49402474da9420" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 41187 } } }
I20260620 18:37:50.992916 1305 sys_catalog.cc:458] T 00000000000000000000000000000000 P b12a2d40d57a46939f49402474da9420 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:37:50.996667 1301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.954887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60612 (local address 127.10.131.60:41187)
0620 18:37:50.955091 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:50.955096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:50.965415 (+ 10319us) server_negotiation.cc:407] Connection header received
0620 18:37:50.965597 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:50.965603 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:50.965691 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:50.965788 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:50.978859 (+ 13071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.979940 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.982635 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.983623 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.983746 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:50.988185 (+ 4439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:50.988195 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:50.988203 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:50.988251 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:50.992301 (+ 4050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:50.993495 (+ 1194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:50.993503 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:50.993510 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:50.993690 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:50.994335 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:50.994355 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:50.994360 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:50.994821 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:50.995112 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:50.995849 (+ 737us) server_negotiation.cc:299] Negotiation successful
0620 18:37:50.996200 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:37:50.999766 1319 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca24be77ad6f4c158ce9c3504fe9a39e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ca24be77ad6f4c158ce9c3504fe9a39e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca24be77ad6f4c158ce9c3504fe9a39e" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 40351 } } peers { permanent_uuid: "e73bcd5c670a4629a2616cb500345400" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 40401 } } peers { permanent_uuid: "b12a2d40d57a46939f49402474da9420" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 41187 } } }
I20260620 18:37:51.000051 1319 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca24be77ad6f4c158ce9c3504fe9a39e [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:51.001111 1336 catalog_manager.cc:1347] Generated new cluster ID: 113812f8a985492393ea7f00339c6654
I20260620 18:37:51.001209 1336 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:37:51.009213 1228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.871666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59952 (local address 127.10.131.61:40401)
0620 18:37:50.871882 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:50.871891 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:50.937798 (+ 65907us) server_negotiation.cc:407] Connection header received
0620 18:37:50.938209 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:50.938216 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:50.938309 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:50.938400 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:50.950713 (+ 12313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.951915 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.964686 (+ 12771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.965625 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.965731 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:50.999162 (+ 33431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:50.999168 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:50.999173 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:50.999213 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:51.002909 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:51.006782 (+ 3873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:51.006788 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:51.006791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:51.006945 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:51.007342 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:51.007346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:51.007350 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:51.007718 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:51.007952 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:51.008604 (+ 652us) server_negotiation.cc:299] Negotiation successful
0620 18:37:51.008830 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:37:51.026119 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:50.871264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57010 (local address 127.10.131.62:40351)
0620 18:37:50.871477 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:50.871486 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:50.955921 (+ 84435us) server_negotiation.cc:407] Connection header received
0620 18:37:50.956999 (+ 1078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:50.957008 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:50.957122 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:50.957212 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:50.974212 (+ 17000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.975328 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.996418 (+ 21090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:50.997442 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:50.997571 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:51.013026 (+ 15455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:37:51.013033 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:37:51.013038 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:37:51.013091 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:37:51.016722 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:51.022903 (+ 6181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:51.022911 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:51.022917 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:51.023089 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:37:51.023925 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:37:51.023930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:37:51.023933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:37:51.024321 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:37:51.024572 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:51.025537 (+ 965us) server_negotiation.cc:299] Negotiation successful
0620 18:37:51.025772 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:37:51.031586 1336 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:37:51.032754 1345 sys_catalog.cc:455] T 00000000000000000000000000000000 P e73bcd5c670a4629a2616cb500345400 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca24be77ad6f4c158ce9c3504fe9a39e. Latest consensus state: current_term: 1 leader_uuid: "ca24be77ad6f4c158ce9c3504fe9a39e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca24be77ad6f4c158ce9c3504fe9a39e" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 40351 } } peers { permanent_uuid: "e73bcd5c670a4629a2616cb500345400" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 40401 } } peers { permanent_uuid: "b12a2d40d57a46939f49402474da9420" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 41187 } } }
I20260620 18:37:51.033071 1345 sys_catalog.cc:458] T 00000000000000000000000000000000 P e73bcd5c670a4629a2616cb500345400 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:37:51.033358 1327 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca24be77ad6f4c158ce9c3504fe9a39e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ca24be77ad6f4c158ce9c3504fe9a39e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca24be77ad6f4c158ce9c3504fe9a39e" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 40351 } } peers { permanent_uuid: "e73bcd5c670a4629a2616cb500345400" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 40401 } } peers { permanent_uuid: "b12a2d40d57a46939f49402474da9420" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 41187 } } }
I20260620 18:37:51.033408 1355 sys_catalog.cc:455] T 00000000000000000000000000000000 P e73bcd5c670a4629a2616cb500345400 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ca24be77ad6f4c158ce9c3504fe9a39e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca24be77ad6f4c158ce9c3504fe9a39e" member_type: VOTER last_known_addr { host: "127.10.131.62" port: 40351 } } peers { permanent_uuid: "e73bcd5c670a4629a2616cb500345400" member_type: VOTER last_known_addr { host: "127.10.131.61" port: 40401 } } peers { permanent_uuid: "b12a2d40d57a46939f49402474da9420" member_type: VOTER last_known_addr { host: "127.10.131.60" port: 41187 } } }
I20260620 18:37:51.033651 1327 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca24be77ad6f4c158ce9c3504fe9a39e [sys.catalog]: This master's current role is: LEADER
I20260620 18:37:51.033749 1355 sys_catalog.cc:458] T 00000000000000000000000000000000 P e73bcd5c670a4629a2616cb500345400 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:37:51.036772 1336 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:37:51.063295 1336 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca24be77ad6f4c158ce9c3504fe9a39e: Generated new TSK 0
I20260620 18:37:51.064445 1336 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:37:51.132447 1365 catalog_manager.cc:1259] Loaded cluster ID: 113812f8a985492393ea7f00339c6654
I20260620 18:37:51.132560 1365 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e73bcd5c670a4629a2616cb500345400: loading cluster ID for follower catalog manager: success
I20260620 18:37:51.139158 1365 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e73bcd5c670a4629a2616cb500345400: acquiring CA information for follower catalog manager: success
I20260620 18:37:51.145857 1365 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e73bcd5c670a4629a2616cb500345400: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 18:37:51.227079 1154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:37:51.223985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57016 (local address 127.10.131.62:40351)
0620 18:37:51.224191 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:51.224198 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:51.225234 (+ 1036us) server_negotiation.cc:407] Connection header received
0620 18:37:51.225497 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:51.225503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:51.225581 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:51.225651 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:51.226827 (+ 1176us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57016: BlockingRecv error: recv got EOF from 127.0.0.1:57016 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 18:37:51.229130 1228 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:37:51.224528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59960 (local address 127.10.131.61:40401)
0620 18:37:51.224752 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:51.224758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:51.227604 (+ 2846us) server_negotiation.cc:407] Connection header received
0620 18:37:51.227848 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:51.227853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:51.227933 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:51.227999 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:51.228927 (+ 928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59960: BlockingRecv error: recv got EOF from 127.0.0.1:59960 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 18:37:51.235670 1301 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:37:51.233539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60618 (local address 127.10.131.60:41187)
0620 18:37:51.233731 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:51.233737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:51.233989 (+ 252us) server_negotiation.cc:407] Connection header received
0620 18:37:51.234217 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:51.234222 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:51.234300 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:51.234401 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:51.235425 (+ 1024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60618: BlockingRecv error: recv got EOF from 127.0.0.1:60618 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:37:51.287194 1228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:51.267676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59974 (local address 127.10.131.61:40401)
0620 18:37:51.267869 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:51.267879 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:51.279185 (+ 11306us) server_negotiation.cc:407] Connection header received
0620 18:37:51.279733 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:51.279740 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:51.279967 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:51.280104 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:37:51.280791 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.281923 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.284104 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.284816 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.284931 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:51.285697 (+ 766us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:37:51.286368 (+ 671us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:37:51.286488 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:51.286753 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 18:37:51.286849 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 18:37:51.346249 1301 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:37:51.329791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60630 (local address 127.10.131.60:41187)
0620 18:37:51.329971 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:51.329978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:51.344217 (+ 14239us) server_negotiation.cc:407] Connection header received
0620 18:37:51.344490 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:51.344496 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:51.344648 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:51.344768 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:51.345990 (+ 1222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60630: BlockingRecv error: recv got EOF from 127.0.0.1:60630 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:37:51.400735 1228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:51.381873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59988 (local address 127.10.131.61:40401)
0620 18:37:51.382082 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:51.382089 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:51.393410 (+ 11321us) server_negotiation.cc:407] Connection header received
0620 18:37:51.394222 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:51.394228 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:51.394354 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:51.394430 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:37:51.395159 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.396319 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.398347 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.399040 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.399139 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:51.399831 (+ 692us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:37:51.399991 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:37:51.400063 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:51.400317 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 18:37:51.400411 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260620 18:37:51.446183 1301 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:37:51.434249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60644 (local address 127.10.131.60:41187)
0620 18:37:51.434457 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:51.434464 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:51.444305 (+ 9841us) server_negotiation.cc:407] Connection header received
0620 18:37:51.444537 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:51.444543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:51.444641 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:51.444758 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:51.445946 (+ 1188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60644: BlockingRecv error: recv got EOF from 127.0.0.1:60644 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:37:51.465943 1228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:51.458892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60000 (local address 127.10.131.61:40401)
0620 18:37:51.459080 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:51.459086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:51.459629 (+ 543us) server_negotiation.cc:407] Connection header received
0620 18:37:51.459851 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:51.459858 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:51.459959 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:51.460078 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:37:51.460564 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.461755 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.463769 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.464441 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.464576 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:51.465101 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:37:51.465245 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:37:51.465347 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:51.465504 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 18:37:51.465601 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 18:37:51.479099 1301 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:37:51.473318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60652 (local address 127.10.131.60:41187)
0620 18:37:51.473557 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:51.473566 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:51.477391 (+ 3825us) server_negotiation.cc:407] Connection header received
0620 18:37:51.477561 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:51.477567 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:51.477682 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:51.477809 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:51.478855 (+ 1046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60652: BlockingRecv error: recv got EOF from 127.0.0.1:60652 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 18:37:51.499959 1228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:51.490917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60012 (local address 127.10.131.61:40401)
0620 18:37:51.491111 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:51.491118 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:51.493442 (+ 2324us) server_negotiation.cc:407] Connection header received
0620 18:37:51.493668 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:51.493674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:51.493782 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:51.493897 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:37:51.494406 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.495656 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.497757 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.498514 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.498647 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:51.499124 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:37:51.499286 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:37:51.499382 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:51.499568 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 18:37:51.499672 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 18:37:51.513145 1301 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:37:51.510599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60664 (local address 127.10.131.60:41187)
0620 18:37:51.510902 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:51.510911 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:51.511479 (+ 568us) server_negotiation.cc:407] Connection header received
0620 18:37:51.511687 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:51.511693 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:51.511801 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:51.511911 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:51.512889 (+ 978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60664: BlockingRecv error: recv got EOF from 127.0.0.1:60664 (error 108)
Metrics: {"server-negotiator.queue_time_us":178}
I20260620 18:37:51.532485 1228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:51.525146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60026 (local address 127.10.131.61:40401)
0620 18:37:51.525359 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:51.525366 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:51.525921 (+ 555us) server_negotiation.cc:407] Connection header received
0620 18:37:51.526131 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:51.526137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:51.526237 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:51.526345 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:37:51.526792 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.528027 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.530071 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.530833 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.530983 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:51.531605 (+ 622us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:37:51.531794 (+ 189us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:37:51.531896 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:51.532081 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 18:37:51.532162 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260620 18:37:51.543067 1301 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:37:51.539669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60666 (local address 127.10.131.60:41187)
0620 18:37:51.539848 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:51.539854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:51.541460 (+ 1606us) server_negotiation.cc:407] Connection header received
0620 18:37:51.541656 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:51.541662 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:51.541761 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:51.541867 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:51.542775 (+ 908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60666: BlockingRecv error: recv got EOF from 127.0.0.1:60666 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:37:51.574126 1228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:51.566944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60042 (local address 127.10.131.61:40401)
0620 18:37:51.567127 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:51.567133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:51.567691 (+ 558us) server_negotiation.cc:407] Connection header received
0620 18:37:51.567911 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:51.567917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:51.568015 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:51.568119 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:37:51.568640 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.569875 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.571892 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.572560 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.572704 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:51.573270 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:37:51.573420 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:37:51.573521 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:51.573689 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 18:37:51.573786 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 18:37:51.582885 1301 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:37:51.580666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60668 (local address 127.10.131.60:41187)
0620 18:37:51.580854 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:51.580862 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:51.581416 (+ 554us) server_negotiation.cc:407] Connection header received
0620 18:37:51.581587 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:51.581593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:51.581706 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:51.581810 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:51.582649 (+ 839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60668: BlockingRecv error: recv got EOF from 127.0.0.1:60668 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:37:51.625535 1228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:51.618838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60056 (local address 127.10.131.61:40401)
0620 18:37:51.619051 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:51.619059 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:51.619518 (+ 459us) server_negotiation.cc:407] Connection header received
0620 18:37:51.619715 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:51.619723 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:51.619826 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:51.619937 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:37:51.620340 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.621516 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.623385 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.624041 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.624185 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:51.624592 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:37:51.624726 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:37:51.624817 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:51.625076 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 18:37:51.625179 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 18:37:51.634019 1301 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:37:51.631721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60678 (local address 127.10.131.60:41187)
0620 18:37:51.631878 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:51.631882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:51.632433 (+ 551us) server_negotiation.cc:407] Connection header received
0620 18:37:51.632633 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:51.632637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:51.632728 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:51.632803 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:51.633721 (+ 918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60678: BlockingRecv error: recv got EOF from 127.0.0.1:60678 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 18:37:51.706832 1228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:51.701632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60058 (local address 127.10.131.61:40401)
0620 18:37:51.701873 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:51.701879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:51.702220 (+ 341us) server_negotiation.cc:407] Connection header received
0620 18:37:51.702466 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:51.702470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:51.702551 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:51.702667 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:37:51.703102 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.703826 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.705196 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.705592 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.705728 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:51.706121 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:37:51.706224 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:37:51.706326 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:51.706481 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 18:37:51.706559 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260620 18:37:51.714241 1301 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:37:51.712199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60688 (local address 127.10.131.60:41187)
0620 18:37:51.712423 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:51.712429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:51.712704 (+ 275us) server_negotiation.cc:407] Connection header received
0620 18:37:51.712975 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:51.712979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:51.713070 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:51.713150 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:37:51.713907 (+ 757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60688: BlockingRecv error: recv got EOF from 127.0.0.1:60688 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:37:51.755098 1316 catalog_manager.cc:1259] Loaded cluster ID: 113812f8a985492393ea7f00339c6654
I20260620 18:37:51.755234 1316 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b12a2d40d57a46939f49402474da9420: loading cluster ID for follower catalog manager: success
I20260620 18:37:51.757750 1316 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b12a2d40d57a46939f49402474da9420: acquiring CA information for follower catalog manager: success
I20260620 18:37:51.759474 1316 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b12a2d40d57a46939f49402474da9420: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:37:51.851229 1228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:51.846341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60072 (local address 127.10.131.61:40401)
0620 18:37:51.846515 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:51.846520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:51.846909 (+ 389us) server_negotiation.cc:407] Connection header received
0620 18:37:51.847109 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:51.847113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:51.847182 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:51.847269 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:37:51.847695 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.848396 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.849750 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.850158 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.850253 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:51.850649 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:37:51.850759 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:37:51.850840 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:51.850954 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 18:37:51.851017 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:37:51.862112 1301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:51.856679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60696 (local address 127.10.131.60:41187)
0620 18:37:51.856922 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:51.856928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:51.857231 (+ 303us) server_negotiation.cc:407] Connection header received
0620 18:37:51.857405 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:51.857410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:51.857553 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:51.857672 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:37:51.858085 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.858764 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.860024 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.860433 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.860556 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:51.861033 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:37:51.861526 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:37:51.861674 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:51.861778 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 18:37:51.861864 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:37:51.875283 1228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:51.867964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60084 (local address 127.10.131.61:40401)
0620 18:37:51.868215 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:51.868223 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:51.869401 (+ 1178us) server_negotiation.cc:407] Connection header received
0620 18:37:51.869622 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:51.869628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:51.869721 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:51.869770 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:37:51.870520 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.871491 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.873376 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.873962 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.874098 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:51.874553 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:37:51.874675 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:37:51.874762 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:51.874917 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 18:37:51.874988 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:37:51.876004 1470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:51.868035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57028 (local address 127.10.131.62:40351)
0620 18:37:51.868974 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:51.868982 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:51.869007 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:37:51.869172 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:51.869178 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:51.869413 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:51.869574 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:37:51.870095 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.871065 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.873051 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.873583 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.873703 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:51.874632 (+ 929us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:37:51.875036 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:37:51.875275 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:51.875636 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 18:37:51.875712 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":787,"thread_start_us":691,"threads_started":1}
I20260620 18:37:51.876606 1301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:37:51.869555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60708 (local address 127.10.131.60:41187)
0620 18:37:51.869706 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 18:37:51.869714 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:37:51.870493 (+ 779us) server_negotiation.cc:407] Connection header received
0620 18:37:51.870730 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:37:51.870735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:37:51.870819 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:37:51.870911 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:37:51.871509 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.872541 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.874588 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:37:51.875201 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:37:51.875318 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:37:51.875890 (+ 572us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:37:51.875996 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:37:51.876045 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 18:37:51.876292 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 18:37:51.876377 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 18:37:51.880443 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 1083
I20260620 18:37:51.881800 1145 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:51.993230 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 1083
I20260620 18:37:52.013650 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 1157
I20260620 18:37:52.014717 1219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:52.119654 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 1157
I20260620 18:37:52.139870 10764 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0dhafo/build/asan/bin/kudu pid 1231
I20260620 18:37:52.141036 1293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:37:52.246582 10764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0dhafo/build/asan/bin/kudu with pid 1231
2026-06-20T18:37:52Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4360 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9264 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (358116 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST